Repository navigation
ci: run check and ci modules in concurrent lanes - #6617
Conversation
Quest-Claim: 6877545fdbc639aa
`just check` and `just ci` ran every selected module one after another. They now run in lanes, MOQ_CHECK_JOBS (default 3) at a time, slowest first. Modules that share mutable state stay serial in one lane: the root Bun workspace (js, drafts, markdown, gh), and the venv, the target/debug moq-ffi library, and cpp/moq/build (carla, py, kt, swift, go, cpp, obs_compile). Concurrent lanes buffer their output and print it when they finish; the first failure cancels the running lanes' process groups, starts no more, and names its phase. fix stays serial. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dart builds moq-ffi with its build hook's filtered environment and checks the hook then finds that build fresh. Any other Cargo build in between reruns host build scripts that track the environment, such as bindgen's BINDGEN_EXTRA_CLANG_ARGS, so the hook rebuilt and the check failed whenever another lane compiled beside it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Outcome: the quest's work is done, and the PR stays a draft for two maintainer decisions, listed in the body: the Measured on a loaded local host: a moq-net plus moq-ffi entry takes 312 s serial and 232 s with 3 lanes on average, the best pair going from 294 s to 182 s. Two races found while measuring are fixed by lane grouping, and a third by running dart alone. The dart leak test is load sensitive on its own; that is recorded as a follow-up. (Written by Claude Opus 5.5) |
# Conflicts: # quest/m0/ci/parallel-modules.md
…s env The quest asks that an injected 'just gh test' failure end the run before the Rust build finishes and leave no module running. Fail gh's tests once rs's tests and kt's check are both blocked, so the run ends only if both lanes are cancelled. Drop the runner's GITHUB_ACTIONS and GITHUB_STEP_SUMMARY from the test's environment: under Actions the failure annotation goes to stdout and the report appends to the real job summary. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
obs-parallel is unblocked and takes its bound from MOQ_CHECK_JOBS. python-setup's rebuild sample predates parallel modules, where py's test now runs right after its check in the same lane. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 18 minutes. View limit details
No actionable comments were generated in the recent review.
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4816395dcc
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A serial run (MOQ_CHECK_JOBS=1, and every 'just fix') left its lane in the terminal's process group, so SIGTERM reached only the lane shell, which defers its trap until the phase exits: the workload and the dispatcher kept running. Give every lane its own process group. A phase's subshell also died on TERM without reaping its children, so a cancelled run could return while they were still exiting. Trap TERM there too, so the subshell exits once its phase does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # quest/m0/ci/README.md # quest/m0/ci/nightly-levers.md
|
Merge summary: merged moq-dev/moq#6639 is now on (Written by Claude Opus 5.5) |
Completes
quest/m0/ci/parallel-modules.md:just checkandjust cirun their selected modules concurrently instead of one after another, with the same selection and fail-fast behavior.What changed
sh/dispatch.shgroups the selected modules into lanes and runsMOQ_CHECK_JOBSlanes at once (default 3), slowest first. Each module runs its check and then its tests (rs, js, py, gh) inside its lane.just fixstays serial, since its fixers rewrite files the other fixers read.Modules that share mutable state share a lane and run in module order. Each grouping comes from a race I reproduced:
bun install --frozen-lockfilerelinks the root workspace even when nothing changed, so concurrent installs fail withEEXIST(5 of 5 concurrent no-op installs failed).uv sync --no-install-workspaceuninstalls py's editablematurin developbuild from the shared.venv(reproduced with a two-member uv workspace). py buildstarget/debug/libmoq_ffi.sowithtesting,openh264, while kt, go, and cpp read theopenh264build from that same path after Cargo exits. kt even runs a secondcargo run uniffi-bindgen --libraryagainst it. cpp and obs_compile also configure and install the samecpp/moq/build/{package,prefix}.sh/dart/generate.shbuilds moq-ffi with the build hook's filtered environment, and the check then fails if the hook rebuilds. Host build scripts intarget/debugare shared across every build. Any full-environment Cargo build in between reruns the ones that track the environment, so the hook rebuilt bindgen, moq-vaapi, moq-video, and moq-ffi. Cargo's fingerprint log showsEnvVarChanged { name: "BINDGEN_EXTRA_CLANG_ARGS" }. I reproduced this with the rs test phase alone running besidejust dart check; dart alone passed. So dart waits for the other lanes to finish, and nothing starts beside it.Output and failure:
::group::per phase on GitHub Actions and==> check rsheaders locally.MOQ_CHECK_JOBS=1streams in order, like before.set -m), serial ones too, and a phase's subshell exits only once its processes do. The first failure cancels the running lanes' whole trees and starts no more, then prints the failure last and names it (ci: check kt failed, plus::error::on Actions). Ctrl-C and SIGTERM cancel the same way, and the run returns only after the cancelled trees exit.(failed)and(cancelled), and end with the run's wall time, because concurrent rows overlap.sh/dispatch-query.test.pygains the quest's injected failure: rs's tests and kt's check block until killed, andjust gh testfails only once both are running. It asserts that the run exits with gh's status and namestest gh, kills both blocked processes, stops kt's lane before go, and starts no lane after the failure. So a cheap failure ends the run before the Rust build finishes. The test also drops the runner'sGITHUB_ACTIONSandGITHUB_STEP_SUMMARY, which would move the failure annotation to stdout and append to the real job summary. A second test sends SIGTERM to a run blocked incheck rs, at 1 and 3 jobs, and asserts it exits 143, recordscheck rs (cancelled), and leaves the blocked process dead.In the queue run the quest cites, the bun lane (js, drafts, markdown, gh) sums to about 113 s and the rs lane to 240 s, so gh's failure would have ended the entry in about 2 minutes instead of 16.
Measurements
These were taken locally on a 24-core host shared with other agents, so the numbers are noisy. The 1-minute load average ran 10 to 70 during these runs. Each run appends a new comment to
rs/moq-net/src/lib.rsandrs/moq-ffi/src/lib.rs, then runsMOQ_STRICT=1 NEXTEST_PROFILE=ci sh/dispatch.sh ci HEAD. That selects the rs dependents of moq-net, every binding, and the repository lints, like a multi-language queue entry. Caches were warm.main)Mean: 312 s before, 232 s after with 3 lanes (about 26% less). On the quieter pair (b) it is 38% less. 4 lanes was not clearly better than 3, so the default stays 3.
The win in the queue depends on rs and the ffi lane overlapping. Queue entries that select both spend 150 to 420 s in
test rsand 400 to 500 s in the bindings, so I expect about 30% off them (870 s down to about 580 s). The quest estimated 45%, and dart running alone is most of the gap.Public API and wire
No public API or wire change.
MOQ_CHECK_JOBSis a new developer environment variable read bysh/dispatch.sh.Checks
main. Its carla check fails onmainitself, since moq-dev/moq#6595 removedTrackProducer.flushbutdemo/carla/simstill calls it; moq-dev/moq#6639 fixes that, and this PR's CI needs it first. The local checks below ran with that fix applied.python3 sh/dispatch-query.test.py(90 runs underGITHUB_ACTIONS=true, no failures),python3 sh/js/workspace.test.py, shellcheck, and shfmt pass.quest checkpasses.just check(every module, since dispatch changed) passes withnix flake checkstubbed out, exceptcheck dart's load-sensitive leak test (see follow-ups), which then passed 2 of 3 standalone runs at load average 40. The flake stub is needed because this host's AppArmorapparmor_restrict_unprivileged_userns=1refuses theunsharethe runner image build needs (unshare: write failed /proc/self/uid_map), unrelated to this change. An unstubbed run hit that failure, and the cancel path ran for real: it cancelled the rs and ffi lanes mid-test and reportedcheck flake failed.obs-parallel.md(now unblocked) andpython-setup.md, whose rebuild sample predates py's check and test sharing a lane.just test interop --all(no wire, FFI, or gateway change).Decisions
Settled with the maintainer:
MOQ_CHECK_JOBS, default 3, ignored byjust fix.build.build-dir, so dart can run beside other lanes, is a follow-up.(Written by Claude Opus 5.5)
Follow-ups (suggested quests)
dart/moq_ffi/test/leak_test.dart's "a non-null optional argument is released" case fails about 20 to 40% of runs standalone on a loaded host: RSS grows 45 to 63 MB against a 32 MB bound.MALLOC_ARENA_MAX=1does not help (4 of 10 failed). Each call throws aMoqExceptionwhose message echoes the 64 KiB reference, so the RSS likely includes Dart garbage the GC has not collected yet, not a native leak. It did not fail in the last 30 failed queue runs, since the runner was otherwise idle. Running dart alone keeps the queue there.local-gate.md's measurements.🤖 Generated with Claude Code
(Written by Claude Opus 5.5)