Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion quest/m0/ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,6 @@ The historical samples locate candidates; they are not controlled benchmarks.
- [Require the release gate](/quest/m0/ci/release-gates.md) - condition: the next cut carries `release-gate.yml`; then release PRs require it
- [Share local check capacity](/quest/m0/ci/local-gate.md) - gate expensive phases and bound internal fanout
- [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the queue builds two or more entries at once instead of one
- [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - `just ci` overlaps its selected modules
- [Share Python check and test preparation](/quest/m0/ci/python-setup.md) - prepare editable bindings once per combined CI pass while preserving standalone commands
- [Parallelize OBS source checks](/quest/m0/ci/obs-parallel.md) - check independent translation units concurrently when OBS is on the critical path
- [Measure nightly compiler levers](/quest/m0/ci/nightly-levers.md) - the parallel frontend and Cranelift, adopted only if the numbers beat their risks
Expand Down
3 changes: 2 additions & 1 deletion quest/m0/ci/local-gate.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ also bounds its own worker fanout so it does not overwhelm the desktop.
must not deadlock or bypass the cap. Cheap selection and docs-only checks
should remain cheap. Verify cancellation leaves no owned workload running.
- Reuse mbx for Rust compilation. Measure nextest thread/group limits and Bun
workspace fanout for local runs; do not blindly enable mbx test scheduling
workspace fanout for local runs, and the module lanes `sh/dispatch.sh`
runs at once (`MOQ_CHECK_JOBS`); do not blindly enable mbx test scheduling
where queued time would consume nextest's timeout budget.
- Benchmark representative solo and concurrent checkouts, recording wall time,
CPU time, worker counts, cache state, and other host load. Compare total work
Expand Down
1 change: 0 additions & 1 deletion quest/m0/ci/nightly-levers.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,4 +34,3 @@ Public API: none. Wire: none.
## Required

- [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the structural baseline
- [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - the structural baseline
9 changes: 3 additions & 6 deletions quest/m0/ci/obs-parallel.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,16 +25,13 @@ entry.
- Finish `sh/obs/includes.sh` and its generated C++ package preparation before
starting concurrent syntax checks. Preserve all source/test coverage,
compiler flags, per-source diagnostics, and the aggregate failure result.
- Take concurrency from the bound parallel modules and local admission share;
do not create a second scheduling policy or unbounded fanout.
- Take concurrency from the module lane bound (`MOQ_CHECK_JOBS` in
`sh/dispatch.sh`) and local admission share; do not create a second
scheduling policy or unbounded fanout.
- Verify a failing source makes the command fail, all sources are checked, and
interruption reaps children. Compare serial and parallel runs on the same
inputs, reporting header preparation separately from compilation.

## Required

- [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - decides whether OBS stays on the critical path, and owns the fanout bound

## Related

- [Share local check capacity](/quest/m0/ci/local-gate.md) - coordinate internal fanout with the shared local command budget
31 changes: 0 additions & 31 deletions quest/m0/ci/parallel-modules.md

This file was deleted.

10 changes: 6 additions & 4 deletions quest/m0/ci/python-setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,11 @@ Decided 2026-10-23: remove repeated Python preparation. Keep existing coverage
and the `testing,openh264` feature contract.

- `py/justfile` currently synchronizes the environment and runs `_develop`
in both recipes. `sh/dispatch.sh` separates them with other binding checks
that also build moq-ffi through Cargo with other features.
in both recipes. The sample below predates parallel modules, when
`sh/dispatch.sh` separated them with other binding checks that also build
moq-ffi through Cargo with other features. py's test now runs right after
its check in the same lane; if the second preparation no longer rebuilds,
delete this quest.
- In [queue run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979),
the second preparation rebuilt the UniFFI generator for 43.70 seconds,
in addition to native build and installation work. Repeated compilation
Expand All @@ -23,8 +26,7 @@ and the `testing,openh264` feature contract.
after Rust, binding, feature, or dependency changes, and stop on preparation
failure. Prefer the smallest recipe/dispatch refactor that achieves this.
- Other languages use different features and build environments. This quest
does not unify every native artifact; running modules concurrently belongs
to [parallel modules](/quest/m0/ci/parallel-modules.md).
does not unify every native artifact.
- Verify one preparation in the combined pass, standalone checks and tests
from a fresh environment, and rebuilds after relevant input changes.
Measure combined Python and full selected-gate time before and after on
Expand Down
71 changes: 66 additions & 5 deletions sh/dispatch-query.test.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,10 @@ def setUp(self):
(self.bin / command).symlink_to(shutil.which(command))
self.env = dict(os.environ, PATH=str(self.bin), GIT_CONFIG_NOSYSTEM="1",
GIT_CONFIG_GLOBAL=os.devnull, MOQ_STRICT="1")
for key in ("MOQ_DEV_SHELL", "MOQ_ALLOW_HOST", "GITHUB_BASE_REF"):
# The runner's own Actions variables would move the run's error
# annotations to stdout and append to the real job summary.
for key in ("MOQ_DEV_SHELL", "MOQ_ALLOW_HOST", "MOQ_CHECK_JOBS", "GITHUB_BASE_REF",
"GITHUB_ACTIONS", "GITHUB_STEP_SUMMARY"):
self.env.pop(key, None)
self.git("init", "-q", "-b", "main")
self.git("config", "user.name", "test")
Expand Down Expand Up @@ -173,7 +176,7 @@ def test_platform_is_only_a_query(self):
self.assertEqual(result.returncode, 2)
self.assertEqual(result.stdout, "")

def run_ci(self, base="origin/main"):
def stub_ci(self, just=""):
for tool in ("bun", "python3", "cargo", "wasm-bindgen", "jq", "shellcheck",
"shfmt", "taplo", "nixfmt", "actionlint", "quest", "nix",
"c++", "clang-format", "cmake", "dart", "dash", "gersemi", "go",
Expand All @@ -182,9 +185,9 @@ def run_ci(self, base="origin/main"):
path = self.bin / tool
path.write_text("#!/bin/sh\nexit 0\n")
path.chmod(0o755)
just = self.bin / "just"
just.write_text('#!/usr/bin/env bash\nprintf "%s\\n" "$*" >>"$CALLS"\n')
just.chmod(0o755)
path = self.bin / "just"
path.write_text('#!/usr/bin/env bash\nprintf "%s\\n" "$*" >>"$CALLS"\n' + just)
path.chmod(0o755)
nix = self.bin / "nix"
nix.write_text('#!/usr/bin/env bash\nprintf "nix %s\\n" "$*" >>"$CALLS"\n')
nix.chmod(0o755)
Expand All @@ -195,11 +198,69 @@ def run_ci(self, base="origin/main"):
path = self.repo / "sh" / script
path.write_text("#!/bin/sh\nexit 0\n")
path.chmod(0o755)

def run_ci(self, base="origin/main"):
self.stub_ci()
result = subprocess.run(["bash", "sh/dispatch.sh", "ci", base],
cwd=self.repo, env=self.env, text=True, capture_output=True)
self.assertEqual(result.returncode, 0, result.stderr)
return (self.root / "calls").read_text().splitlines()

def test_failure_cancels_running_lanes_and_names_its_phase(self):
# rs's tests and kt's check run until killed, and gh's tests fail only
# once both are running, so the run ends only if the failure cancels
# both lanes.
blocked = {}
for name, call in (("rs", '"rs test-changed "*'), ("kt", '"kt check"')):
started = self.root / f"{name}.started"
os.mkfifo(started)
pid = self.root / f"{name}.pid"
blocked[name] = (call, started, pid)
stub = "".join(f' {call}) echo $$ >"{pid}"; echo >"{started}"; exec sleep 600 ;;\n'
for call, started, pid in blocked.values())
reads = "; ".join(f'read -r <"{started}"' for _, started, _ in blocked.values())
self.stub_ci(f'case "$*" in\n{stub} "gh test") {reads}; exit 3 ;;\nesac\n')
self.change("rs/moq-ffi/src/lib.rs")
result = subprocess.run(["bash", "sh/dispatch.sh", "ci", "origin/main"], cwd=self.repo,
env=self.env, text=True, capture_output=True, timeout=60)
self.assertEqual(result.returncode, 3, result.stderr)
self.assertIn("ci: test gh failed", result.stderr)
self.assertRegex(result.stderr, r"\d+ test gh \(failed\)")
self.assertRegex(result.stderr, r"\d+ test rs \(cancelled\)")
self.assertRegex(result.stderr, r"\d+ check kt \(cancelled\)")
for _, _, pid in blocked.values():
with self.assertRaises(ProcessLookupError):
os.kill(int(pid.read_text()), 0)
# A cancelled lane starts no more of its modules, and no lane starts
# after the failure.
calls = (self.root / "calls").read_text().splitlines()
self.assertIn("py test", calls)
self.assertNotIn("go check", calls)
self.assertNotIn("dart check", calls)

def test_terminating_the_run_stops_the_running_phase(self):
started = self.root / "started"
os.mkfifo(started)
pid = self.root / "pid"
self.stub_ci(f'''case "$*" in
"rs lint-changed "*) echo $$ >"{pid}"; echo >"{started}"; exec sleep 600 ;;
esac
''')
self.change("rs/moq-clock/src/lib.rs")
for jobs in ("1", "3"):
with self.subTest(jobs=jobs):
run = subprocess.Popen(["bash", "sh/dispatch.sh", "ci", "origin/main"],
cwd=self.repo, env=dict(self.env, MOQ_CHECK_JOBS=jobs),
text=True, stdout=subprocess.DEVNULL,
stderr=subprocess.PIPE)
started.read_text()
run.terminate()
_, stderr = run.communicate(timeout=60)
self.assertEqual(run.returncode, 143, stderr)
self.assertRegex(stderr, r"\d+ check rs \(cancelled\)")
with self.assertRaises(ProcessLookupError):
os.kill(int(pid.read_text()), 0)

def test_ci_runs_one_lint_preamble_and_one_test_pass(self):
self.change("rs/moq-clock/src/lib.rs")
calls = self.run_ci()
Expand Down
Loading
Loading