Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion quest/m0/ci/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,6 @@ The historical samples locate candidates; they are not controlled benchmarks.
- [Share local check capacity](/quest/m0/ci/local-gate.md) - gate expensive phases and bound internal fanout
- [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the queue builds two or more entries at once instead of one
- [Share Python check and test preparation](/quest/m0/ci/python-setup.md) - prepare editable bindings once per combined CI pass while preserving standalone commands
- [Parallelize OBS source checks](/quest/m0/ci/obs-parallel.md) - check independent translation units concurrently when OBS is on the critical path
- [Measure nightly compiler levers](/quest/m0/ci/nightly-levers.md) - the parallel frontend and Cranelift, adopted only if the numbers beat their risks
- [Bump to the manifest fix](/quest/m0/ci/mbx-bump.md) - condition: an mbx release contains jdx/mr-boxington#798; then bump the Rust cache's pin
- [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - measure the hosted Nightly jobs' hits, restore time, and expiry once the bump lands
Expand Down
3 changes: 2 additions & 1 deletion quest/m0/ci/local-gate.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,8 @@ also bounds its own worker fanout so it does not overwhelm the desktop.
should remain cheap. Verify cancellation leaves no owned workload running.
- Reuse mbx for Rust compilation. Measure nextest thread/group limits and Bun
workspace fanout for local runs, and the module lanes `sh/dispatch.sh`
runs at once (`MOQ_CHECK_JOBS`); do not blindly enable mbx test scheduling
runs at once (`MOQ_CHECK_JOBS`, which also bounds the sources
`sh/obs/compile.sh` checks at once); do not blindly enable mbx test scheduling
where queued time would consume nextest's timeout budget.
- Benchmark representative solo and concurrent checkouts, recording wall time,
CPU time, worker counts, cache state, and other host load. Compare total work
Expand Down
37 changes: 0 additions & 37 deletions quest/m0/ci/obs-parallel.md

This file was deleted.

69 changes: 57 additions & 12 deletions sh/obs/compile.sh
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,14 @@ for pkgs in "Qt6Widgets Qt6Gui Qt6Core" "libavcodec libavutil libswscale libswre
fi
done

# Sources check in parallel under the same bound as the module lanes
# sh/dispatch.sh runs at once, and with the same default.
limit=${MOQ_CHECK_JOBS:-3}
if [[ ! "$limit" =~ ^[1-9][0-9]*$ ]]; then
echo "error: MOQ_CHECK_JOBS must be a positive integer, not '$limit'" >&2
exit 2
fi

# Assign first so a failure in includes.sh still aborts under `set -e`,
# then split on newlines only: an include path may contain spaces.
includes_raw=$("$here/includes.sh")
Expand All @@ -47,19 +55,56 @@ while IFS= read -r flag; do includes+=("$flag"); done <<<"$includes_raw"
qt=$(pkg-config --cflags Qt6Widgets Qt6Gui Qt6Core)
ffmpeg=$(pkg-config --cflags libavcodec libavutil libswscale libswresample)

# MOQ_VERSION_STRING only reaches a label in the dock, so any value
# type-checks the same; CMake stamps the real moq-cpp version.
logs=$(mktemp -d)
declare -A running=()
# Each check runs in its own process group, so a cancel reaches the compiler
# proper that the driver forks, not only the driver.
trap 'for pid in "${!running[@]}"; do kill -TERM -- "-$pid" 2>/dev/null || true; done; wait; rm -rf "$logs"' EXIT
trap 'exit 130' INT
trap 'exit 143' TERM

# A failing source does not stop the rest, so one run reports every broken
# file. Each source's diagnostics print whole once it finishes, rather than
# interleaved with the checks beside it.
sources=(src/*.cpp test/*.cpp)
next=0
status=0
for source in src/*.cpp test/*.cpp; do
# shellcheck disable=SC2086
# -Werror: a warning that only prints is never fixed.
# -Wno-unused-command-line-argument: the nix compiler wrapper injects
# link flags, which a syntax-only run reports as unused, once per file.
# $qt and $ffmpeg stay unquoted on purpose: pkg-config hands back one
# space-separated string, and splitting it is the intended reading.
if ! "$cxx" -std=c++17 -fsyntax-only -Werror -Wno-unused-command-line-argument \
"${includes[@]}" $qt $ffmpeg \
-DMOQ_FRONTEND_ENABLED -DMOQ_VERSION_STRING='"0.0.0"' "$source"; then
while true; do
while ((next < ${#sources[@]} && ${#running[@]} < limit)); do
set -m
# shellcheck disable=SC2086
# MOQ_VERSION_STRING only reaches a label in the dock, so any value
# type-checks the same; CMake stamps the real moq-cpp version.
# -Werror: a warning that only prints is never fixed.
# -Wno-unused-command-line-argument: the nix compiler wrapper injects
# link flags, which a syntax-only run reports as unused, once per file.
# $qt and $ffmpeg stay unquoted on purpose: pkg-config hands back one
# space-separated string, and splitting it is the intended reading.
# The job is a subshell around the compiler, so it exits even when the
# compiler is killed: bash 5.2 drops a job that a signal ends before
# `wait -n` collects it. The trailing exit keeps bash from exec'ing the
# compiler in the subshell's place.
(
"$cxx" -std=c++17 -fsyntax-only -Werror -Wno-unused-command-line-argument \
"${includes[@]}" $qt $ffmpeg \
-DMOQ_FRONTEND_ENABLED -DMOQ_VERSION_STRING='"0.0.0"' "${sources[next]}"
exit
) </dev/null >"$logs/$next" 2>&1 &
set +m
running[$!]=$next
next=$((next + 1))
done
((${#running[@]})) || break

result=0
wait -n -p pid "${!running[@]}" || result=$?
index=${running[$pid]}
unset "running[$pid]"
cat "$logs/$index" >&2
# Named here because a compiler killed outright, such as by the OOM killer
# under concurrent checks, prints nothing that says which source it was.
if ((result)); then
echo "error: ${sources[index]} failed with status $result" >&2
status=1
fi
done
Expand Down
Loading