diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index ac4277f2016..5a17251d99f 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -91,7 +91,6 @@ The historical samples locate candidates; they are not controlled benchmarks. - [Require the release gate](/quest/m0/ci/release-gates.md) - condition: the next cut carries `release-gate.yml`; then release PRs require it - [Share local check capacity](/quest/m0/ci/local-gate.md) - gate expensive phases and bound internal fanout - [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the queue builds two or more entries at once instead of one -- [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - `just ci` overlaps its selected modules - [Share Python check and test preparation](/quest/m0/ci/python-setup.md) - prepare editable bindings once per combined CI pass while preserving standalone commands - [Parallelize OBS source checks](/quest/m0/ci/obs-parallel.md) - check independent translation units concurrently when OBS is on the critical path - [Measure nightly compiler levers](/quest/m0/ci/nightly-levers.md) - the parallel frontend and Cranelift, adopted only if the numbers beat their risks diff --git a/quest/m0/ci/local-gate.md b/quest/m0/ci/local-gate.md index e640c236061..f16aba11b5d 100644 --- a/quest/m0/ci/local-gate.md +++ b/quest/m0/ci/local-gate.md @@ -15,7 +15,8 @@ also bounds its own worker fanout so it does not overwhelm the desktop. must not deadlock or bypass the cap. Cheap selection and docs-only checks should remain cheap. Verify cancellation leaves no owned workload running. - Reuse mbx for Rust compilation. Measure nextest thread/group limits and Bun - workspace fanout for local runs; do not blindly enable mbx test scheduling + workspace fanout for local runs, and the module lanes `sh/dispatch.sh` + runs at once (`MOQ_CHECK_JOBS`); do not blindly enable mbx test scheduling where queued time would consume nextest's timeout budget. - Benchmark representative solo and concurrent checkouts, recording wall time, CPU time, worker counts, cache state, and other host load. Compare total work diff --git a/quest/m0/ci/nightly-levers.md b/quest/m0/ci/nightly-levers.md index e67cb6f452d..46c8a7dced0 100644 --- a/quest/m0/ci/nightly-levers.md +++ b/quest/m0/ci/nightly-levers.md @@ -34,4 +34,3 @@ Public API: none. Wire: none. ## Required - [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the structural baseline -- [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - the structural baseline diff --git a/quest/m0/ci/obs-parallel.md b/quest/m0/ci/obs-parallel.md index 473c36223e5..25c441ae2d3 100644 --- a/quest/m0/ci/obs-parallel.md +++ b/quest/m0/ci/obs-parallel.md @@ -25,16 +25,13 @@ entry. - Finish `sh/obs/includes.sh` and its generated C++ package preparation before starting concurrent syntax checks. Preserve all source/test coverage, compiler flags, per-source diagnostics, and the aggregate failure result. -- Take concurrency from the bound parallel modules and local admission share; - do not create a second scheduling policy or unbounded fanout. +- Take concurrency from the module lane bound (`MOQ_CHECK_JOBS` in + `sh/dispatch.sh`) and local admission share; do not create a second + scheduling policy or unbounded fanout. - Verify a failing source makes the command fail, all sources are checked, and interruption reaps children. Compare serial and parallel runs on the same inputs, reporting header preparation separately from compilation. -## Required - -- [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - decides whether OBS stays on the critical path, and owns the fanout bound - ## Related - [Share local check capacity](/quest/m0/ci/local-gate.md) - coordinate internal fanout with the shared local command budget diff --git a/quest/m0/ci/parallel-modules.md b/quest/m0/ci/parallel-modules.md deleted file mode 100644 index 78f0261424d..00000000000 --- a/quest/m0/ci/parallel-modules.md +++ /dev/null @@ -1,31 +0,0 @@ -# [M] Run the queue's modules in parallel - -## Goal - -`just ci` runs its selected modules concurrently instead of one after another, -with the same modules selected and the same failure behavior, and a -multi-language entry finishes measurably faster. - -## Plan - -Decided with the maintainer on 2026-10-23. Measured 2026-10-23: `sh/dispatch.sh` -runs every selected module serially. On multi-language entries (863 to 876 s), -the cpp, dart, py, kt, and obs modules take about 400 to 500 s that could -overlap the Rust build, an estimated 45% off those entries. - -- Bound concurrency so modules do not starve each other, and keep each - module's output readable when they interleave. -- A failure still fails the run and names its module, and cancels the - modules still running, so a cheap check that fails ends the entry in its own - time. In [queue run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979) - the tooling tests failed after 18 s, but only after about 16 minutes of - modules ahead of them. Verify an injected `just gh test` failure ends the run - before the Rust build finishes, and that cancellation leaves no module - running. Decided 2026-10-23: this replaces a separate quest to move cheap - checks ahead of builds in the serial loop. - -Public API: none. Wire: none. - -## Related - -- [Share local check capacity](/quest/m0/ci/local-gate.md) - the same fanout bound applies locally diff --git a/quest/m0/ci/python-setup.md b/quest/m0/ci/python-setup.md index 7258403c16f..64d556fa223 100644 --- a/quest/m0/ci/python-setup.md +++ b/quest/m0/ci/python-setup.md @@ -12,8 +12,11 @@ Decided 2026-10-23: remove repeated Python preparation. Keep existing coverage and the `testing,openh264` feature contract. - `py/justfile` currently synchronizes the environment and runs `_develop` - in both recipes. `sh/dispatch.sh` separates them with other binding checks - that also build moq-ffi through Cargo with other features. + in both recipes. The sample below predates parallel modules, when + `sh/dispatch.sh` separated them with other binding checks that also build + moq-ffi through Cargo with other features. py's test now runs right after + its check in the same lane; if the second preparation no longer rebuilds, + delete this quest. - In [queue run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979), the second preparation rebuilt the UniFFI generator for 43.70 seconds, in addition to native build and installation work. Repeated compilation @@ -23,8 +26,7 @@ and the `testing,openh264` feature contract. after Rust, binding, feature, or dependency changes, and stop on preparation failure. Prefer the smallest recipe/dispatch refactor that achieves this. - Other languages use different features and build environments. This quest - does not unify every native artifact; running modules concurrently belongs - to [parallel modules](/quest/m0/ci/parallel-modules.md). + does not unify every native artifact. - Verify one preparation in the combined pass, standalone checks and tests from a fresh environment, and rebuilds after relevant input changes. Measure combined Python and full selected-gate time before and after on diff --git a/sh/dispatch-query.test.py b/sh/dispatch-query.test.py index 1bd05ddaaf7..db82b7c4a6c 100644 --- a/sh/dispatch-query.test.py +++ b/sh/dispatch-query.test.py @@ -30,7 +30,10 @@ def setUp(self): (self.bin / command).symlink_to(shutil.which(command)) self.env = dict(os.environ, PATH=str(self.bin), GIT_CONFIG_NOSYSTEM="1", GIT_CONFIG_GLOBAL=os.devnull, MOQ_STRICT="1") - for key in ("MOQ_DEV_SHELL", "MOQ_ALLOW_HOST", "GITHUB_BASE_REF"): + # The runner's own Actions variables would move the run's error + # annotations to stdout and append to the real job summary. + for key in ("MOQ_DEV_SHELL", "MOQ_ALLOW_HOST", "MOQ_CHECK_JOBS", "GITHUB_BASE_REF", + "GITHUB_ACTIONS", "GITHUB_STEP_SUMMARY"): self.env.pop(key, None) self.git("init", "-q", "-b", "main") self.git("config", "user.name", "test") @@ -173,7 +176,7 @@ def test_platform_is_only_a_query(self): self.assertEqual(result.returncode, 2) self.assertEqual(result.stdout, "") - def run_ci(self, base="origin/main"): + def stub_ci(self, just=""): for tool in ("bun", "python3", "cargo", "wasm-bindgen", "jq", "shellcheck", "shfmt", "taplo", "nixfmt", "actionlint", "quest", "nix", "c++", "clang-format", "cmake", "dart", "dash", "gersemi", "go", @@ -182,9 +185,9 @@ def run_ci(self, base="origin/main"): path = self.bin / tool path.write_text("#!/bin/sh\nexit 0\n") path.chmod(0o755) - just = self.bin / "just" - just.write_text('#!/usr/bin/env bash\nprintf "%s\\n" "$*" >>"$CALLS"\n') - just.chmod(0o755) + path = self.bin / "just" + path.write_text('#!/usr/bin/env bash\nprintf "%s\\n" "$*" >>"$CALLS"\n' + just) + path.chmod(0o755) nix = self.bin / "nix" nix.write_text('#!/usr/bin/env bash\nprintf "nix %s\\n" "$*" >>"$CALLS"\n') nix.chmod(0o755) @@ -195,11 +198,69 @@ def run_ci(self, base="origin/main"): path = self.repo / "sh" / script path.write_text("#!/bin/sh\nexit 0\n") path.chmod(0o755) + + def run_ci(self, base="origin/main"): + self.stub_ci() result = subprocess.run(["bash", "sh/dispatch.sh", "ci", base], cwd=self.repo, env=self.env, text=True, capture_output=True) self.assertEqual(result.returncode, 0, result.stderr) return (self.root / "calls").read_text().splitlines() + def test_failure_cancels_running_lanes_and_names_its_phase(self): + # rs's tests and kt's check run until killed, and gh's tests fail only + # once both are running, so the run ends only if the failure cancels + # both lanes. + blocked = {} + for name, call in (("rs", '"rs test-changed "*'), ("kt", '"kt check"')): + started = self.root / f"{name}.started" + os.mkfifo(started) + pid = self.root / f"{name}.pid" + blocked[name] = (call, started, pid) + stub = "".join(f' {call}) echo $$ >"{pid}"; echo >"{started}"; exec sleep 600 ;;\n' + for call, started, pid in blocked.values()) + reads = "; ".join(f'read -r <"{started}"' for _, started, _ in blocked.values()) + self.stub_ci(f'case "$*" in\n{stub} "gh test") {reads}; exit 3 ;;\nesac\n') + self.change("rs/moq-ffi/src/lib.rs") + result = subprocess.run(["bash", "sh/dispatch.sh", "ci", "origin/main"], cwd=self.repo, + env=self.env, text=True, capture_output=True, timeout=60) + self.assertEqual(result.returncode, 3, result.stderr) + self.assertIn("ci: test gh failed", result.stderr) + self.assertRegex(result.stderr, r"\d+ test gh \(failed\)") + self.assertRegex(result.stderr, r"\d+ test rs \(cancelled\)") + self.assertRegex(result.stderr, r"\d+ check kt \(cancelled\)") + for _, _, pid in blocked.values(): + with self.assertRaises(ProcessLookupError): + os.kill(int(pid.read_text()), 0) + # A cancelled lane starts no more of its modules, and no lane starts + # after the failure. + calls = (self.root / "calls").read_text().splitlines() + self.assertIn("py test", calls) + self.assertNotIn("go check", calls) + self.assertNotIn("dart check", calls) + + def test_terminating_the_run_stops_the_running_phase(self): + started = self.root / "started" + os.mkfifo(started) + pid = self.root / "pid" + self.stub_ci(f'''case "$*" in + "rs lint-changed "*) echo $$ >"{pid}"; echo >"{started}"; exec sleep 600 ;; +esac +''') + self.change("rs/moq-clock/src/lib.rs") + for jobs in ("1", "3"): + with self.subTest(jobs=jobs): + run = subprocess.Popen(["bash", "sh/dispatch.sh", "ci", "origin/main"], + cwd=self.repo, env=dict(self.env, MOQ_CHECK_JOBS=jobs), + text=True, stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE) + started.read_text() + run.terminate() + _, stderr = run.communicate(timeout=60) + self.assertEqual(run.returncode, 143, stderr) + self.assertRegex(stderr, r"\d+ check rs \(cancelled\)") + with self.assertRaises(ProcessLookupError): + os.kill(int(pid.read_text()), 0) + def test_ci_runs_one_lint_preamble_and_one_test_pass(self): self.change("rs/moq-clock/src/lib.rs") calls = self.run_ci() diff --git a/sh/dispatch.sh b/sh/dispatch.sh index ee43d44bd4b..a543ea6364a 100755 --- a/sh/dispatch.sh +++ b/sh/dispatch.sh @@ -1,5 +1,7 @@ #!/usr/bin/env bash # Run `just check`, `just fix`, or `just ci` over one resolved branch diff. +# check and ci run MOQ_CHECK_JOBS lanes of modules at once (default 3); 1 runs +# them in order and streams their output. # `--query` prints the selection before CI installs its toolchain and caches. # `ci-platform` exists only as a query for the native platform workflows. set -euo pipefail @@ -175,11 +177,42 @@ declare -A compiles=( [fix]='rs' ) +# Lanes start in this order, so the slowest come first. dart runs alone (see +# below), so it waits for the rest. case "$action" in - check | ci) modules=(js workers moq_sh drafts rs bench carla quest drill py kt swift go dart cpp obs_compile obs flake ts_paths markdown shell large_files toml nix justfile gh) ;; + check | ci) modules=(rs carla py kt swift go cpp obs_compile js drafts markdown gh flake bench drill workers moq_sh quest obs ts_paths shell large_files toml nix justfile dart) ;; fix) modules=(js rs py carla dart obs markdown shell toml nix justfile) ;; esac +# Modules that share mutable state run in one lane, in module order; every +# other module is a lane of its own, and lanes run concurrently. +declare -A lane=( + # `bun install` relinks the root workspace's packages even when nothing + # changed, so a concurrent install or import of them fails. + [js]=bun + [drafts]=bun + [markdown]=bun + [gh]=bun + # carla's `uv sync` uninstalls py's editable build from the shared venv. + # py builds target/debug/libmoq_ffi.so with its test features, and the + # other bindings read their default build from that path after Cargo + # exits. cpp and obs_compile also share cpp/moq/build. + [carla]=ffi + [py]=ffi + [kt]=ffi + [swift]=ffi + [go]=ffi + [cpp]=ffi + [obs_compile]=ffi +) + +# Lanes that run with no other lane beside them. dart builds moq-ffi with the +# filtered environment its build hook gets, then checks that the hook finds +# that build fresh. Any other Cargo build in between reruns the host build +# scripts in target/debug that track the environment, such as bindgen's +# BINDGEN_EXTRA_CLANG_ARGS, and so rebuilds what the hook then rebuilds again. +declare -A alone=([dart]=1) + selected=() missing=() for module in "${modules[@]}"; do @@ -246,48 +279,15 @@ files() { git ls-files -z --cached --others --exclude-standard -- "$@" } -# Each phase's wall time, printed when the run ends, pass or fail, so the slow -# module of a check or a queue run is named without reading log timestamps. -# GitHub Actions also gets it in the job summary. -labels=() -seconds=() -current= -began=0 -begin() { - current=$1 - began=$SECONDS -} -finish() { - [[ -n "$current" ]] || return 0 - labels+=("$current${1:+ ($1)}") - seconds+=("$((SECONDS - began))") - current= -} -report() { - finish failed - ((${#labels[@]})) || return 0 - echo "$action: seconds per module" >&2 - for i in "${!labels[@]}"; do - printf '%6d %s\n' "${seconds[$i]}" "${labels[$i]}" >&2 - done - [[ -n "${GITHUB_STEP_SUMMARY:-}" ]] || return 0 - { - printf '| %s module | Seconds |\n| --- | ---: |\n' "$action" - for i in "${!labels[@]}"; do - printf '| %s | %d |\n' "${labels[$i]}" "${seconds[$i]}" - done - } >>"$GITHUB_STEP_SUMMARY" -} -trap 'report; rm -f "$changed" "$js_changed"' EXIT - -for module in "${selected[@]}"; do - begin "$verb $module" - case "$verb:$module" in +# One phase of one module. The bindings run their tests in their check; rs, +# js, py, and gh test right after it, in the same lane. +step() { + case "$1:$2" in check:js) just js check-changed "$js_list" ;; check:workers) just js workers ;; check:moq_sh) just infra moq-sh test ;; - # The preamble keeps docs and target-specific lints; the test build - # below enforces default-feature clippy without a second compile. + # The check keeps docs and target-specific lints; the test phase + # enforces default-feature clippy without a second compile. check:rs) just rs lint-changed "$rs_list" ;; check:bench) python3 bench/compare.test.py @@ -306,6 +306,10 @@ for module in "${selected[@]}"; do check:toml) RUST_LOG=error taplo format --check ;; check:nix) files '*.nix' | xargs -0 nixfmt --check ;; check:justfile) files justfile '*/justfile' | xargs -0 -n1 just --fmt --check --justfile ;; + test:rs) just rs test-changed "$rs_list" ;; + test:js) just js test-changed "$js_list" ;; + test:py) just py test ;; + test:gh) just gh test ;; fix:rs) just rs fix-changed "$rs_list" ;; fix:markdown) bun install @@ -314,26 +318,173 @@ for module in "${selected[@]}"; do fix:toml) RUST_LOG=error taplo format ;; fix:nix) files '*.nix' | xargs -0 nixfmt ;; fix:justfile) files justfile '*/justfile' | xargs -0 -n1 just --fmt --justfile ;; - *:shell) sh/shell.sh "$verb" ;; - *) just "$module" "$verb" ;; + *:shell) sh/shell.sh "$1" ;; + *) just "$2" "$1" ;; esac - finish +} + +# Lanes run at most this many at a time. fix stays serial: its fixers rewrite +# files the others read, such as taplo under `cargo clippy --fix`. +limit=${MOQ_CHECK_JOBS:-3} +[[ "$action" != fix ]] || limit=1 +if [[ ! "$limit" =~ ^[1-9][0-9]*$ ]]; then + echo "error: MOQ_CHECK_JOBS must be a positive integer, not '$limit'" >&2 + exit 2 +fi + +lanes=() +declare -A members=() +for module in "${selected[@]}"; do + name=${lane[$module]:-$module} + [[ -v "members[$name]" ]] || lanes+=("$name") + members[$name]+=" $module" done -# The bindings run their tests in the preamble; these suites run once here. -# gh's tooling tests run on every diff: path selection would let a broken -# selector skip the very tests that catch it. -if [[ "$verb" == check ]]; then - for module in "${selected[@]}"; do - case "$module" in - rs | js | py | gh) begin "test $module" ;; - esac - case "$module" in - rs) just rs test-changed "$rs_list" ;; - js) just js test-changed "$js_list" ;; - py) just py test ;; - gh) just gh test ;; - esac - finish +github() { + [[ -n "${GITHUB_ACTIONS:-}" ]] +} + +# Run one lane's modules in order, appending each phase's seconds to the lane's +# times file. The lane stops at its first failure. A TERM, from the scheduler +# cancelling the run, records the phase it cut short. +run_lane() { + times=$1 + shift + current= + trap 'cancelled; exit 143' TERM + for module in "$@"; do + phases=("$verb") + # gh's tooling tests run on every diff: path selection would let a + # broken selector skip the very tests that catch it. + if [[ "$verb" == check ]]; then + case "$module" in + rs | js | py | gh) phases+=(test) ;; + esac + fi + for phase in "${phases[@]}"; do + current="$phase $module" + began=$SECONDS + if github; then echo "::group::$current"; else echo "==> $current"; fi + # A plain statement, so errexit stays on inside the subshell. + set +e + ( + set -e + # A subshell dies on TERM at once, orphaning the phase's + # processes. Trapped, it exits once they do, so a cancel + # returns only after the whole tree has exited. + trap 'exit 143' TERM + step "$phase" "$module" + ) + status=$? + set -e + ! github || echo "::endgroup::" + if ((status)); then + record failed + return "$status" + fi + record + current= + done done +} +record() { + echo "$((SECONDS - began)) $current${1:+ ($1)}" >>"$times" +} +cancelled() { + [[ -n "$current" ]] || return 0 + ! github || echo "::endgroup::" + record cancelled +} + +# Each phase's wall time, printed when the run ends, pass or fail, so the slow +# module of a check or a queue run is named without reading log timestamps. +# GitHub Actions also gets it in the job summary. Concurrent phases overlap, so +# the run's own wall time comes last. +report() { + local name row seconds label rows=() + for name in "${lanes[@]:0:next}"; do + [[ ! -f "$logs/$name.times" ]] || mapfile -t -O "${#rows[@]}" rows <"$logs/$name.times" + done + ((${#rows[@]})) || return 0 + rows+=("$SECONDS wall time") + echo "$action: seconds per module" >&2 + for row in "${rows[@]}"; do + read -r seconds label <<<"$row" + printf '%6d %s\n' "$seconds" "$label" >&2 + done + [[ -n "${GITHUB_STEP_SUMMARY:-}" ]] || return 0 + { + printf '| %s module | Seconds |\n| --- | ---: |\n' "$action" + for row in "${rows[@]}"; do + read -r seconds label <<<"$row" + printf '| %s | %d |\n' "$label" "$seconds" + done + } >>"$GITHUB_STEP_SUMMARY" +} + +# Stop every running lane, whose process group holds its whole tree. +cancel() { + local pid + for pid in "${!running[@]}"; do + kill -TERM -- "-$pid" 2>/dev/null || true + done +} + +logs=$(mktemp -d) +declare -A running=() +lone= +next=0 +failed= +failed_status=0 +trap 'cancel; wait; report; rm -rf "$logs" "$changed" "$js_changed"' EXIT +trap 'exit 130' INT +trap 'exit 143' TERM + +# Each lane runs in its own process group, so a cancel reaches its whole tree; +# a lane shell defers its own TERM trap until the running phase exits. +# Concurrent lanes buffer their output and print it whole when they finish; a +# serial run streams instead. The first failure cancels the running lanes and +# starts no more, as a serial run stops at its first failure. +set -m +while true; do + while [[ -z "$failed" && -z "$lone" ]] && ((next < ${#lanes[@]} && ${#running[@]} < limit)); do + name=${lanes[next]} + if [[ -v "alone[$name]" ]] && ((${#running[@]})); then + break + fi + next=$((next + 1)) + read -ra lane_modules <<<"${members[$name]}" + if ((limit == 1)); then + run_lane "$logs/$name.times" "${lane_modules[@]}" &2 + run_lane "$logs/$name.times" "${lane_modules[@]}" "$logs/$name.log" 2>&1 & + fi + running[$!]=$name + [[ ! -v "alone[$name]" ]] || lone=$! + done + ((${#running[@]})) || break + + status=0 + wait -n -p pid "${!running[@]}" || status=$? + name=${running[$pid]} + unset "running[$pid]" + [[ "$pid" != "$lone" ]] || lone= + if ((status)) && [[ -z "$failed" ]]; then + failed=$name + failed_status=$status + cancel + continue + fi + ((limit == 1)) || cat "$logs/$name.log" + [[ -z "$failed" || "$status" != 143 ]] || echo "$action: cancelled ${members[$name]# }" >&2 +done + +# The failure prints last, after what it cancelled. +if [[ -n "$failed" ]]; then + ((limit == 1)) || cat "$logs/$failed.log" + phase=$(sed -n 's/^[0-9]* \(.*\) (failed)$/\1/p' "$logs/$failed.times") + phase=${phase:-${members[$failed]# }} + if github; then echo "::error::$action: $phase failed"; else echo "$action: $phase failed" >&2; fi + exit "$failed_status" fi