From a364d4ae9344675c147a25a4a59d0db67ccfa3ae Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 23:11:09 -0700 Subject: [PATCH 1/6] chore(quest): claim quest/m0/ci/parallel-modules.md Quest-Claim: 6877545fdbc639aa From ab893264bd4136a77311aa8ce2009fbd899a3416 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 23:59:11 -0700 Subject: [PATCH 2/6] ci: run check and ci modules in concurrent lanes `just check` and `just ci` ran every selected module one after another. They now run in lanes, MOQ_CHECK_JOBS (default 3) at a time, slowest first. Modules that share mutable state stay serial in one lane: the root Bun workspace (js, drafts, markdown, gh), and the venv, the target/debug moq-ffi library, and cpp/moq/build (carla, py, kt, swift, go, cpp, obs_compile). Concurrent lanes buffer their output and print it when they finish; the first failure cancels the running lanes' process groups, starts no more, and names its phase. fix stays serial. Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/README.md | 1 - quest/m0/ci/local-gate.md | 3 +- quest/m0/ci/nightly-levers.md | 1 - quest/m0/ci/parallel-modules.md | 24 ---- sh/dispatch-query.test.py | 38 ++++- sh/dispatch.sh | 248 ++++++++++++++++++++++++-------- 6 files changed, 226 insertions(+), 89 deletions(-) delete mode 100644 quest/m0/ci/parallel-modules.md diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 2b699f61031..e54d8c169e3 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -91,7 +91,6 @@ The historical samples locate candidates; they are not controlled benchmarks. - [Require the release gate](/quest/m0/ci/release-gates.md) - condition: the next cut carries `release-gate.yml`; then release PRs require it - [Share local check capacity](/quest/m0/ci/local-gate.md) - gate expensive phases and bound internal fanout - [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the queue builds two or more entries at once instead of one -- [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - `just ci` overlaps its selected modules - [One nextest build in the queue](/quest/m0/ci/one-test-build.md) - `test rs` compiles once instead of three times - [Measure nightly compiler levers](/quest/m0/ci/nightly-levers.md) - the parallel frontend and Cranelift, adopted only if the numbers beat their risks - [Bump to the manifest fix](/quest/m0/ci/mbx-bump.md) - condition: an mbx release contains jdx/mr-boxington#798; then bump the Rust cache's pin diff --git a/quest/m0/ci/local-gate.md b/quest/m0/ci/local-gate.md index e640c236061..f16aba11b5d 100644 --- a/quest/m0/ci/local-gate.md +++ b/quest/m0/ci/local-gate.md @@ -15,7 +15,8 @@ also bounds its own worker fanout so it does not overwhelm the desktop. must not deadlock or bypass the cap. Cheap selection and docs-only checks should remain cheap. Verify cancellation leaves no owned workload running. - Reuse mbx for Rust compilation. Measure nextest thread/group limits and Bun - workspace fanout for local runs; do not blindly enable mbx test scheduling + workspace fanout for local runs, and the module lanes `sh/dispatch.sh` + runs at once (`MOQ_CHECK_JOBS`); do not blindly enable mbx test scheduling where queued time would consume nextest's timeout budget. - Benchmark representative solo and concurrent checkouts, recording wall time, CPU time, worker counts, cache state, and other host load. Compare total work diff --git a/quest/m0/ci/nightly-levers.md b/quest/m0/ci/nightly-levers.md index 663b523afea..cc42e171560 100644 --- a/quest/m0/ci/nightly-levers.md +++ b/quest/m0/ci/nightly-levers.md @@ -34,5 +34,4 @@ Public API: none. Wire: none. ## Required - [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the structural baseline -- [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - the structural baseline - [One nextest build in the queue](/quest/m0/ci/one-test-build.md) - the build this would speed up diff --git a/quest/m0/ci/parallel-modules.md b/quest/m0/ci/parallel-modules.md deleted file mode 100644 index e0d986107ca..00000000000 --- a/quest/m0/ci/parallel-modules.md +++ /dev/null @@ -1,24 +0,0 @@ -# [M] Run the queue's modules in parallel - -## Goal - -`just ci` runs its selected modules concurrently instead of one after another, -with the same modules selected and the same failure behavior, and a -multi-language entry finishes measurably faster. - -## Plan - -Decided with the maintainer on 2026-10-23. Measured 2026-10-23: `sh/dispatch.sh` -runs every selected module serially. On multi-language entries (863 to 876 s), -the cpp, dart, py, kt, and obs modules take about 400 to 500 s that could -overlap the Rust build, an estimated 45% off those entries. - -- Bound concurrency so modules do not starve each other, and keep each - module's output readable when they interleave. -- A failure still fails the run and names its module. - -Public API: none. Wire: none. - -## Related - -- [Share local check capacity](/quest/m0/ci/local-gate.md) - the same fanout bound applies locally diff --git a/sh/dispatch-query.test.py b/sh/dispatch-query.test.py index 1bd05ddaaf7..aebf497b897 100644 --- a/sh/dispatch-query.test.py +++ b/sh/dispatch-query.test.py @@ -173,7 +173,7 @@ def test_platform_is_only_a_query(self): self.assertEqual(result.returncode, 2) self.assertEqual(result.stdout, "") - def run_ci(self, base="origin/main"): + def stub_ci(self, just=""): for tool in ("bun", "python3", "cargo", "wasm-bindgen", "jq", "shellcheck", "shfmt", "taplo", "nixfmt", "actionlint", "quest", "nix", "c++", "clang-format", "cmake", "dart", "dash", "gersemi", "go", @@ -182,9 +182,9 @@ def run_ci(self, base="origin/main"): path = self.bin / tool path.write_text("#!/bin/sh\nexit 0\n") path.chmod(0o755) - just = self.bin / "just" - just.write_text('#!/usr/bin/env bash\nprintf "%s\\n" "$*" >>"$CALLS"\n') - just.chmod(0o755) + path = self.bin / "just" + path.write_text('#!/usr/bin/env bash\nprintf "%s\\n" "$*" >>"$CALLS"\n' + just) + path.chmod(0o755) nix = self.bin / "nix" nix.write_text('#!/usr/bin/env bash\nprintf "nix %s\\n" "$*" >>"$CALLS"\n') nix.chmod(0o755) @@ -195,11 +195,41 @@ def run_ci(self, base="origin/main"): path = self.repo / "sh" / script path.write_text("#!/bin/sh\nexit 0\n") path.chmod(0o755) + + def run_ci(self, base="origin/main"): + self.stub_ci() result = subprocess.run(["bash", "sh/dispatch.sh", "ci", base], cwd=self.repo, env=self.env, text=True, capture_output=True) self.assertEqual(result.returncode, 0, result.stderr) return (self.root / "calls").read_text().splitlines() + def test_failure_cancels_running_lanes_and_names_its_phase(self): + # rs's tests run until killed, and kt fails only once they are running, + # so the run ends only if the failure cancels the other lane. + started = self.root / "started" + os.mkfifo(started) + pid = self.root / "pid" + self.stub_ci(f'''case "$*" in + "rs test-changed "*) echo $$ >"{pid}"; echo >"{started}"; exec sleep 600 ;; + "kt check") read -r <"{started}"; exit 3 ;; +esac +''') + self.change("rs/moq-ffi/src/lib.rs") + result = subprocess.run(["bash", "sh/dispatch.sh", "ci", "origin/main"], cwd=self.repo, + env=dict(self.env, MOQ_CHECK_JOBS="2"), text=True, + capture_output=True, timeout=60) + self.assertEqual(result.returncode, 3, result.stderr) + self.assertIn("ci: check kt failed", result.stderr) + self.assertRegex(result.stderr, r"\d+ check kt \(failed\)") + self.assertRegex(result.stderr, r"\d+ test rs \(cancelled\)") + with self.assertRaises(ProcessLookupError): + os.kill(int(pid.read_text()), 0) + # kt's lane stops at the failure, and no other lane starts after it. + calls = (self.root / "calls").read_text().splitlines() + self.assertIn("py test", calls) + self.assertNotIn("go check", calls) + self.assertNotIn("dart check", calls) + def test_ci_runs_one_lint_preamble_and_one_test_pass(self): self.change("rs/moq-clock/src/lib.rs") calls = self.run_ci() diff --git a/sh/dispatch.sh b/sh/dispatch.sh index ee43d44bd4b..1dc998da803 100755 --- a/sh/dispatch.sh +++ b/sh/dispatch.sh @@ -1,5 +1,7 @@ #!/usr/bin/env bash # Run `just check`, `just fix`, or `just ci` over one resolved branch diff. +# check and ci run MOQ_CHECK_JOBS lanes of modules at once (default 3); 1 runs +# them in order and streams their output. # `--query` prints the selection before CI installs its toolchain and caches. # `ci-platform` exists only as a query for the native platform workflows. set -euo pipefail @@ -175,11 +177,34 @@ declare -A compiles=( [fix]='rs' ) +# Lanes start in this order, so the slowest come first. case "$action" in - check | ci) modules=(js workers moq_sh drafts rs bench carla quest drill py kt swift go dart cpp obs_compile obs flake ts_paths markdown shell large_files toml nix justfile gh) ;; + check | ci) modules=(rs carla py kt swift go cpp obs_compile dart js drafts markdown gh flake bench drill workers moq_sh quest obs ts_paths shell large_files toml nix justfile) ;; fix) modules=(js rs py carla dart obs markdown shell toml nix justfile) ;; esac +# Modules that share mutable state run in one lane, in module order; every +# other module is a lane of its own, and lanes run concurrently. +declare -A lane=( + # `bun install` relinks the root workspace's packages even when nothing + # changed, so a concurrent install or import of them fails. + [js]=bun + [drafts]=bun + [markdown]=bun + [gh]=bun + # carla's `uv sync` uninstalls py's editable build from the shared venv. + # py builds target/debug/libmoq_ffi.so with its test features, and the + # other bindings read their default build from that path after Cargo + # exits. cpp and obs_compile also share cpp/moq/build. + [carla]=ffi + [py]=ffi + [kt]=ffi + [swift]=ffi + [go]=ffi + [cpp]=ffi + [obs_compile]=ffi +) + selected=() missing=() for module in "${modules[@]}"; do @@ -246,48 +271,15 @@ files() { git ls-files -z --cached --others --exclude-standard -- "$@" } -# Each phase's wall time, printed when the run ends, pass or fail, so the slow -# module of a check or a queue run is named without reading log timestamps. -# GitHub Actions also gets it in the job summary. -labels=() -seconds=() -current= -began=0 -begin() { - current=$1 - began=$SECONDS -} -finish() { - [[ -n "$current" ]] || return 0 - labels+=("$current${1:+ ($1)}") - seconds+=("$((SECONDS - began))") - current= -} -report() { - finish failed - ((${#labels[@]})) || return 0 - echo "$action: seconds per module" >&2 - for i in "${!labels[@]}"; do - printf '%6d %s\n' "${seconds[$i]}" "${labels[$i]}" >&2 - done - [[ -n "${GITHUB_STEP_SUMMARY:-}" ]] || return 0 - { - printf '| %s module | Seconds |\n| --- | ---: |\n' "$action" - for i in "${!labels[@]}"; do - printf '| %s | %d |\n' "${labels[$i]}" "${seconds[$i]}" - done - } >>"$GITHUB_STEP_SUMMARY" -} -trap 'report; rm -f "$changed" "$js_changed"' EXIT - -for module in "${selected[@]}"; do - begin "$verb $module" - case "$verb:$module" in +# One phase of one module. The bindings run their tests in their check; rs, +# js, py, and gh test right after it, in the same lane. +step() { + case "$1:$2" in check:js) just js check-changed "$js_list" ;; check:workers) just js workers ;; check:moq_sh) just infra moq-sh test ;; - # The preamble keeps docs and target-specific lints; the test build - # below enforces default-feature clippy without a second compile. + # The check keeps docs and target-specific lints; the test phase + # enforces default-feature clippy without a second compile. check:rs) just rs lint-changed "$rs_list" ;; check:bench) python3 bench/compare.test.py @@ -306,6 +298,10 @@ for module in "${selected[@]}"; do check:toml) RUST_LOG=error taplo format --check ;; check:nix) files '*.nix' | xargs -0 nixfmt --check ;; check:justfile) files justfile '*/justfile' | xargs -0 -n1 just --fmt --check --justfile ;; + test:rs) just rs test-changed "$rs_list" ;; + test:js) just js test-changed "$js_list" ;; + test:py) just py test ;; + test:gh) just gh test ;; fix:rs) just rs fix-changed "$rs_list" ;; fix:markdown) bun install @@ -314,26 +310,162 @@ for module in "${selected[@]}"; do fix:toml) RUST_LOG=error taplo format ;; fix:nix) files '*.nix' | xargs -0 nixfmt ;; fix:justfile) files justfile '*/justfile' | xargs -0 -n1 just --fmt --justfile ;; - *:shell) sh/shell.sh "$verb" ;; - *) just "$module" "$verb" ;; + *:shell) sh/shell.sh "$1" ;; + *) just "$2" "$1" ;; esac - finish +} + +# Lanes run at most this many at a time. fix stays serial: its fixers rewrite +# files the others read, such as taplo under `cargo clippy --fix`. +limit=${MOQ_CHECK_JOBS:-3} +[[ "$action" != fix ]] || limit=1 +if [[ ! "$limit" =~ ^[1-9][0-9]*$ ]]; then + echo "error: MOQ_CHECK_JOBS must be a positive integer, not '$limit'" >&2 + exit 2 +fi + +lanes=() +declare -A members=() +for module in "${selected[@]}"; do + name=${lane[$module]:-$module} + [[ -v "members[$name]" ]] || lanes+=("$name") + members[$name]+=" $module" done -# The bindings run their tests in the preamble; these suites run once here. -# gh's tooling tests run on every diff: path selection would let a broken -# selector skip the very tests that catch it. -if [[ "$verb" == check ]]; then - for module in "${selected[@]}"; do - case "$module" in - rs | js | py | gh) begin "test $module" ;; - esac - case "$module" in - rs) just rs test-changed "$rs_list" ;; - js) just js test-changed "$js_list" ;; - py) just py test ;; - gh) just gh test ;; - esac - finish +github() { + [[ -n "${GITHUB_ACTIONS:-}" ]] +} + +# Run one lane's modules in order, appending each phase's seconds to the lane's +# times file. The lane stops at its first failure. A TERM, from the scheduler +# cancelling the run, records the phase it cut short. +run_lane() { + times=$1 + shift + current= + trap 'cancelled; exit 143' TERM + for module in "$@"; do + phases=("$verb") + # gh's tooling tests run on every diff: path selection would let a + # broken selector skip the very tests that catch it. + if [[ "$verb" == check ]]; then + case "$module" in + rs | js | py | gh) phases+=(test) ;; + esac + fi + for phase in "${phases[@]}"; do + current="$phase $module" + began=$SECONDS + if github; then echo "::group::$current"; else echo "==> $current"; fi + # A plain statement, so errexit stays on inside the subshell. + set +e + ( + set -e + step "$phase" "$module" + ) + status=$? + set -e + ! github || echo "::endgroup::" + if ((status)); then + record failed + return "$status" + fi + record + current= + done + done +} +record() { + echo "$((SECONDS - began)) $current${1:+ ($1)}" >>"$times" +} +cancelled() { + [[ -n "$current" ]] || return 0 + ! github || echo "::endgroup::" + record cancelled +} + +# Each phase's wall time, printed when the run ends, pass or fail, so the slow +# module of a check or a queue run is named without reading log timestamps. +# GitHub Actions also gets it in the job summary. Concurrent phases overlap, so +# the run's own wall time comes last. +report() { + local name row seconds label rows=() + for name in "${lanes[@]:0:next}"; do + [[ ! -f "$logs/$name.times" ]] || mapfile -t -O "${#rows[@]}" rows <"$logs/$name.times" + done + ((${#rows[@]})) || return 0 + rows+=("$SECONDS wall time") + echo "$action: seconds per module" >&2 + for row in "${rows[@]}"; do + read -r seconds label <<<"$row" + printf '%6d %s\n' "$seconds" "$label" >&2 + done + [[ -n "${GITHUB_STEP_SUMMARY:-}" ]] || return 0 + { + printf '| %s module | Seconds |\n| --- | ---: |\n' "$action" + for row in "${rows[@]}"; do + read -r seconds label <<<"$row" + printf '| %s | %d |\n' "$label" "$seconds" + done + } >>"$GITHUB_STEP_SUMMARY" +} + +# Stop every running lane, whose process group holds its whole tree. +cancel() { + local pid + for pid in "${!running[@]}"; do + kill -TERM -- "-$pid" 2>/dev/null || kill -TERM "$pid" 2>/dev/null || true + done +} + +logs=$(mktemp -d) +declare -A running=() +next=0 +failed= +failed_status=0 +trap 'cancel; wait; report; rm -rf "$logs" "$changed" "$js_changed"' EXIT +trap 'exit 130' INT +trap 'exit 143' TERM + +# Concurrent lanes buffer their output and print it whole when they finish, +# each in its own process group so a cancel reaches its whole tree. A serial +# run streams instead, in the terminal's group. The first failure cancels the +# running lanes and starts no more, as a serial run stops at its first failure. +((limit == 1)) || set -m +while true; do + while [[ -z "$failed" ]] && ((next < ${#lanes[@]} && ${#running[@]} < limit)); do + name=${lanes[next]} + next=$((next + 1)) + read -ra lane_modules <<<"${members[$name]}" + if ((limit == 1)); then + run_lane "$logs/$name.times" "${lane_modules[@]}" &2 + run_lane "$logs/$name.times" "${lane_modules[@]}" "$logs/$name.log" 2>&1 & + fi + running[$!]=$name done + ((${#running[@]})) || break + + status=0 + wait -n -p pid "${!running[@]}" || status=$? + name=${running[$pid]} + unset "running[$pid]" + if ((status)) && [[ -z "$failed" ]]; then + failed=$name + failed_status=$status + cancel + continue + fi + ((limit == 1)) || cat "$logs/$name.log" + [[ -z "$failed" || "$status" != 143 ]] || echo "$action: cancelled ${members[$name]# }" >&2 +done + +# The failure prints last, after what it cancelled. +if [[ -n "$failed" ]]; then + ((limit == 1)) || cat "$logs/$failed.log" + phase=$(sed -n 's/^[0-9]* \(.*\) (failed)$/\1/p' "$logs/$failed.times") + phase=${phase:-${members[$failed]# }} + if github; then echo "::error::$action: $phase failed"; else echo "$action: $phase failed" >&2; fi + exit "$failed_status" fi From 3c7f9a262fc80b720792556d94b2f7655c154196 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 00:56:07 -0700 Subject: [PATCH 3/6] ci: run dart's check with no other lane beside it dart builds moq-ffi with its build hook's filtered environment and checks the hook then finds that build fresh. Any other Cargo build in between reruns host build scripts that track the environment, such as bindgen's BINDGEN_EXTRA_CLANG_ARGS, so the hook rebuilt and the check failed whenever another lane compiled beside it. Co-Authored-By: Claude Opus 5.5 --- sh/dispatch.sh | 20 +++++++++++++++++--- 1 file changed, 17 insertions(+), 3 deletions(-) diff --git a/sh/dispatch.sh b/sh/dispatch.sh index 1dc998da803..06312348d1c 100755 --- a/sh/dispatch.sh +++ b/sh/dispatch.sh @@ -177,9 +177,10 @@ declare -A compiles=( [fix]='rs' ) -# Lanes start in this order, so the slowest come first. +# Lanes start in this order, so the slowest come first. dart runs alone (see +# below), so it waits for the rest. case "$action" in - check | ci) modules=(rs carla py kt swift go cpp obs_compile dart js drafts markdown gh flake bench drill workers moq_sh quest obs ts_paths shell large_files toml nix justfile) ;; + check | ci) modules=(rs carla py kt swift go cpp obs_compile js drafts markdown gh flake bench drill workers moq_sh quest obs ts_paths shell large_files toml nix justfile dart) ;; fix) modules=(js rs py carla dart obs markdown shell toml nix justfile) ;; esac @@ -205,6 +206,13 @@ declare -A lane=( [obs_compile]=ffi ) +# Lanes that run with no other lane beside them. dart builds moq-ffi with the +# filtered environment its build hook gets, then checks that the hook finds +# that build fresh. Any other Cargo build in between reruns the host build +# scripts in target/debug that track the environment, such as bindgen's +# BINDGEN_EXTRA_CLANG_ARGS, and so rebuilds what the hook then rebuilds again. +declare -A alone=([dart]=1) + selected=() missing=() for module in "${modules[@]}"; do @@ -420,6 +428,7 @@ cancel() { logs=$(mktemp -d) declare -A running=() +lone= next=0 failed= failed_status=0 @@ -433,8 +442,11 @@ trap 'exit 143' TERM # running lanes and starts no more, as a serial run stops at its first failure. ((limit == 1)) || set -m while true; do - while [[ -z "$failed" ]] && ((next < ${#lanes[@]} && ${#running[@]} < limit)); do + while [[ -z "$failed" && -z "$lone" ]] && ((next < ${#lanes[@]} && ${#running[@]} < limit)); do name=${lanes[next]} + if [[ -v "alone[$name]" ]] && ((${#running[@]})); then + break + fi next=$((next + 1)) read -ra lane_modules <<<"${members[$name]}" if ((limit == 1)); then @@ -444,6 +456,7 @@ while true; do run_lane "$logs/$name.times" "${lane_modules[@]}" "$logs/$name.log" 2>&1 & fi running[$!]=$name + [[ ! -v "alone[$name]" ]] || lone=$! done ((${#running[@]})) || break @@ -451,6 +464,7 @@ while true; do wait -n -p pid "${!running[@]}" || status=$? name=${running[$pid]} unset "running[$pid]" + [[ "$pid" != "$lone" ]] || lone= if ((status)) && [[ -z "$failed" ]]; then failed=$name failed_status=$status From fd4894dd4526625604716ee93097802ed7d05e51 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 07:42:56 -0700 Subject: [PATCH 4/6] test(dispatch): inject the quest's gh test failure and isolate Actions env The quest asks that an injected 'just gh test' failure end the run before the Rust build finishes and leave no module running. Fail gh's tests once rs's tests and kt's check are both blocked, so the run ends only if both lanes are cancelled. Drop the runner's GITHUB_ACTIONS and GITHUB_STEP_SUMMARY from the test's environment: under Actions the failure annotation goes to stdout and the report appends to the real job summary. Co-Authored-By: Claude Opus 5.5 --- sh/dispatch-query.test.py | 44 +++++++++++++++++++++++---------------- 1 file changed, 26 insertions(+), 18 deletions(-) diff --git a/sh/dispatch-query.test.py b/sh/dispatch-query.test.py index aebf497b897..69aee22fba5 100644 --- a/sh/dispatch-query.test.py +++ b/sh/dispatch-query.test.py @@ -30,7 +30,10 @@ def setUp(self): (self.bin / command).symlink_to(shutil.which(command)) self.env = dict(os.environ, PATH=str(self.bin), GIT_CONFIG_NOSYSTEM="1", GIT_CONFIG_GLOBAL=os.devnull, MOQ_STRICT="1") - for key in ("MOQ_DEV_SHELL", "MOQ_ALLOW_HOST", "GITHUB_BASE_REF"): + # The runner's own Actions variables would move the run's error + # annotations to stdout and append to the real job summary. + for key in ("MOQ_DEV_SHELL", "MOQ_ALLOW_HOST", "MOQ_CHECK_JOBS", "GITHUB_BASE_REF", + "GITHUB_ACTIONS", "GITHUB_STEP_SUMMARY"): self.env.pop(key, None) self.git("init", "-q", "-b", "main") self.git("config", "user.name", "test") @@ -204,27 +207,32 @@ def run_ci(self, base="origin/main"): return (self.root / "calls").read_text().splitlines() def test_failure_cancels_running_lanes_and_names_its_phase(self): - # rs's tests run until killed, and kt fails only once they are running, - # so the run ends only if the failure cancels the other lane. - started = self.root / "started" - os.mkfifo(started) - pid = self.root / "pid" - self.stub_ci(f'''case "$*" in - "rs test-changed "*) echo $$ >"{pid}"; echo >"{started}"; exec sleep 600 ;; - "kt check") read -r <"{started}"; exit 3 ;; -esac -''') + # rs's tests and kt's check run until killed, and gh's tests fail only + # once both are running, so the run ends only if the failure cancels + # both lanes. + blocked = {} + for name, call in (("rs", '"rs test-changed "*'), ("kt", '"kt check"')): + started = self.root / f"{name}.started" + os.mkfifo(started) + pid = self.root / f"{name}.pid" + blocked[name] = (call, started, pid) + stub = "".join(f' {call}) echo $$ >"{pid}"; echo >"{started}"; exec sleep 600 ;;\n' + for call, started, pid in blocked.values()) + reads = "; ".join(f'read -r <"{started}"' for _, started, _ in blocked.values()) + self.stub_ci(f'case "$*" in\n{stub} "gh test") {reads}; exit 3 ;;\nesac\n') self.change("rs/moq-ffi/src/lib.rs") result = subprocess.run(["bash", "sh/dispatch.sh", "ci", "origin/main"], cwd=self.repo, - env=dict(self.env, MOQ_CHECK_JOBS="2"), text=True, - capture_output=True, timeout=60) + env=self.env, text=True, capture_output=True, timeout=60) self.assertEqual(result.returncode, 3, result.stderr) - self.assertIn("ci: check kt failed", result.stderr) - self.assertRegex(result.stderr, r"\d+ check kt \(failed\)") + self.assertIn("ci: test gh failed", result.stderr) + self.assertRegex(result.stderr, r"\d+ test gh \(failed\)") self.assertRegex(result.stderr, r"\d+ test rs \(cancelled\)") - with self.assertRaises(ProcessLookupError): - os.kill(int(pid.read_text()), 0) - # kt's lane stops at the failure, and no other lane starts after it. + self.assertRegex(result.stderr, r"\d+ check kt \(cancelled\)") + for _, _, pid in blocked.values(): + with self.assertRaises(ProcessLookupError): + os.kill(int(pid.read_text()), 0) + # A cancelled lane starts no more of its modules, and no lane starts + # after the failure. calls = (self.root / "calls").read_text().splitlines() self.assertIn("py test", calls) self.assertNotIn("go check", calls) From 4816395dcccf7051cbe5beb45f6a7f379978383f Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 08:03:25 -0700 Subject: [PATCH 5/6] quest: drop links to the finished parallel-modules quest obs-parallel is unblocked and takes its bound from MOQ_CHECK_JOBS. python-setup's rebuild sample predates parallel modules, where py's test now runs right after its check in the same lane. Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/obs-parallel.md | 9 +++------ quest/m0/ci/python-setup.md | 10 ++++++---- 2 files changed, 9 insertions(+), 10 deletions(-) diff --git a/quest/m0/ci/obs-parallel.md b/quest/m0/ci/obs-parallel.md index 473c36223e5..25c441ae2d3 100644 --- a/quest/m0/ci/obs-parallel.md +++ b/quest/m0/ci/obs-parallel.md @@ -25,16 +25,13 @@ entry. - Finish `sh/obs/includes.sh` and its generated C++ package preparation before starting concurrent syntax checks. Preserve all source/test coverage, compiler flags, per-source diagnostics, and the aggregate failure result. -- Take concurrency from the bound parallel modules and local admission share; - do not create a second scheduling policy or unbounded fanout. +- Take concurrency from the module lane bound (`MOQ_CHECK_JOBS` in + `sh/dispatch.sh`) and local admission share; do not create a second + scheduling policy or unbounded fanout. - Verify a failing source makes the command fail, all sources are checked, and interruption reaps children. Compare serial and parallel runs on the same inputs, reporting header preparation separately from compilation. -## Required - -- [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - decides whether OBS stays on the critical path, and owns the fanout bound - ## Related - [Share local check capacity](/quest/m0/ci/local-gate.md) - coordinate internal fanout with the shared local command budget diff --git a/quest/m0/ci/python-setup.md b/quest/m0/ci/python-setup.md index 7258403c16f..64d556fa223 100644 --- a/quest/m0/ci/python-setup.md +++ b/quest/m0/ci/python-setup.md @@ -12,8 +12,11 @@ Decided 2026-10-23: remove repeated Python preparation. Keep existing coverage and the `testing,openh264` feature contract. - `py/justfile` currently synchronizes the environment and runs `_develop` - in both recipes. `sh/dispatch.sh` separates them with other binding checks - that also build moq-ffi through Cargo with other features. + in both recipes. The sample below predates parallel modules, when + `sh/dispatch.sh` separated them with other binding checks that also build + moq-ffi through Cargo with other features. py's test now runs right after + its check in the same lane; if the second preparation no longer rebuilds, + delete this quest. - In [queue run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979), the second preparation rebuilt the UniFFI generator for 43.70 seconds, in addition to native build and installation work. Repeated compilation @@ -23,8 +26,7 @@ and the `testing,openh264` feature contract. after Rust, binding, feature, or dependency changes, and stop on preparation failure. Prefer the smallest recipe/dispatch refactor that achieves this. - Other languages use different features and build environments. This quest - does not unify every native artifact; running modules concurrently belongs - to [parallel modules](/quest/m0/ci/parallel-modules.md). + does not unify every native artifact. - Verify one preparation in the combined pass, standalone checks and tests from a fresh environment, and rebuilds after relevant input changes. Measure combined Python and full selected-gate time before and after on From bf6d4aa103f531f49f5ee8b6e94d923614c15f1b Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 08:35:05 -0700 Subject: [PATCH 6/6] fix(dispatch): cancel a serial run's phase and wait for its tree A serial run (MOQ_CHECK_JOBS=1, and every 'just fix') left its lane in the terminal's process group, so SIGTERM reached only the lane shell, which defers its trap until the phase exits: the workload and the dispatcher kept running. Give every lane its own process group. A phase's subshell also died on TERM without reaping its children, so a cancelled run could return while they were still exiting. Trap TERM there too, so the subshell exits once its phase does. Co-Authored-By: Claude Opus 5.5 --- sh/dispatch-query.test.py | 23 +++++++++++++++++++++++ sh/dispatch.sh | 17 +++++++++++------ 2 files changed, 34 insertions(+), 6 deletions(-) diff --git a/sh/dispatch-query.test.py b/sh/dispatch-query.test.py index 69aee22fba5..db82b7c4a6c 100644 --- a/sh/dispatch-query.test.py +++ b/sh/dispatch-query.test.py @@ -238,6 +238,29 @@ def test_failure_cancels_running_lanes_and_names_its_phase(self): self.assertNotIn("go check", calls) self.assertNotIn("dart check", calls) + def test_terminating_the_run_stops_the_running_phase(self): + started = self.root / "started" + os.mkfifo(started) + pid = self.root / "pid" + self.stub_ci(f'''case "$*" in + "rs lint-changed "*) echo $$ >"{pid}"; echo >"{started}"; exec sleep 600 ;; +esac +''') + self.change("rs/moq-clock/src/lib.rs") + for jobs in ("1", "3"): + with self.subTest(jobs=jobs): + run = subprocess.Popen(["bash", "sh/dispatch.sh", "ci", "origin/main"], + cwd=self.repo, env=dict(self.env, MOQ_CHECK_JOBS=jobs), + text=True, stdout=subprocess.DEVNULL, + stderr=subprocess.PIPE) + started.read_text() + run.terminate() + _, stderr = run.communicate(timeout=60) + self.assertEqual(run.returncode, 143, stderr) + self.assertRegex(stderr, r"\d+ check rs \(cancelled\)") + with self.assertRaises(ProcessLookupError): + os.kill(int(pid.read_text()), 0) + def test_ci_runs_one_lint_preamble_and_one_test_pass(self): self.change("rs/moq-clock/src/lib.rs") calls = self.run_ci() diff --git a/sh/dispatch.sh b/sh/dispatch.sh index 06312348d1c..a543ea6364a 100755 --- a/sh/dispatch.sh +++ b/sh/dispatch.sh @@ -369,6 +369,10 @@ run_lane() { set +e ( set -e + # A subshell dies on TERM at once, orphaning the phase's + # processes. Trapped, it exits once they do, so a cancel + # returns only after the whole tree has exited. + trap 'exit 143' TERM step "$phase" "$module" ) status=$? @@ -422,7 +426,7 @@ report() { cancel() { local pid for pid in "${!running[@]}"; do - kill -TERM -- "-$pid" 2>/dev/null || kill -TERM "$pid" 2>/dev/null || true + kill -TERM -- "-$pid" 2>/dev/null || true done } @@ -436,11 +440,12 @@ trap 'cancel; wait; report; rm -rf "$logs" "$changed" "$js_changed"' EXIT trap 'exit 130' INT trap 'exit 143' TERM -# Concurrent lanes buffer their output and print it whole when they finish, -# each in its own process group so a cancel reaches its whole tree. A serial -# run streams instead, in the terminal's group. The first failure cancels the -# running lanes and starts no more, as a serial run stops at its first failure. -((limit == 1)) || set -m +# Each lane runs in its own process group, so a cancel reaches its whole tree; +# a lane shell defers its own TERM trap until the running phase exits. +# Concurrent lanes buffer their output and print it whole when they finish; a +# serial run streams instead. The first failure cancels the running lanes and +# starts no more, as a serial run stops at its first failure. +set -m while true; do while [[ -z "$failed" && -z "$lone" ]] && ((next < ${#lanes[@]} && ${#running[@]} < limit)); do name=${lanes[next]}