Skip to content

ci: run check and ci modules in concurrent lanes - #6617

Merged
kixelated merged 8 commits into
mainfrom
quest/m0/ci/parallel-modules
Oct 24, 2026
Merged

kixelated merged 8 commits into
mainfrom
quest/m0/ci/parallel-modules

Conversation

@kixelated

@kixelated kixelated commented Oct 24, 2026 •

Copy link
Copy Markdown
Collaborator

Completes quest/m0/ci/parallel-modules.md: just check and just ci run their selected modules concurrently instead of one after another, with the same selection and fail-fast behavior.

What changed

sh/dispatch.sh groups the selected modules into lanes and runs MOQ_CHECK_JOBS lanes at once (default 3), slowest first. Each module runs its check and then its tests (rs, js, py, gh) inside its lane. just fix stays serial, since its fixers rewrite files the other fixers read.

Modules that share mutable state share a lane and run in module order. Each grouping comes from a race I reproduced:

  • bun (js, drafts, markdown, gh): bun install --frozen-lockfile relinks the root workspace even when nothing changed, so concurrent installs fail with EEXIST (5 of 5 concurrent no-op installs failed).
  • ffi (carla, py, kt, swift, go, cpp, obs_compile): carla's uv sync --no-install-workspace uninstalls py's editable maturin develop build from the shared .venv (reproduced with a two-member uv workspace). py builds target/debug/libmoq_ffi.so with testing,openh264, while kt, go, and cpp read the openh264 build from that same path after Cargo exits. kt even runs a second cargo run uniffi-bindgen --library against it. cpp and obs_compile also configure and install the same cpp/moq/build/{package,prefix}.
  • dart runs alone: sh/dart/generate.sh builds moq-ffi with the build hook's filtered environment, and the check then fails if the hook rebuilds. Host build scripts in target/debug are shared across every build. Any full-environment Cargo build in between reruns the ones that track the environment, so the hook rebuilt bindgen, moq-vaapi, moq-video, and moq-ffi. Cargo's fingerprint log shows EnvVarChanged { name: "BINDGEN_EXTRA_CLANG_ARGS" }. I reproduced this with the rs test phase alone running beside just dart check; dart alone passed. So dart waits for the other lanes to finish, and nothing starts beside it.

Output and failure:

  • Concurrent lanes buffer their output and print it whole when they finish, with ::group:: per phase on GitHub Actions and ==> check rs headers locally. MOQ_CHECK_JOBS=1 streams in order, like before.
  • Each lane runs in its own process group (set -m), serial ones too, and a phase's subshell exits only once its processes do. The first failure cancels the running lanes' whole trees and starts no more, then prints the failure last and names it (ci: check kt failed, plus ::error:: on Actions). Ctrl-C and SIGTERM cancel the same way, and the run returns only after the cancelled trees exit.
  • The per-module seconds table and job summary keep one row per phase, mark (failed) and (cancelled), and end with the run's wall time, because concurrent rows overlap.

sh/dispatch-query.test.py gains the quest's injected failure: rs's tests and kt's check block until killed, and just gh test fails only once both are running. It asserts that the run exits with gh's status and names test gh, kills both blocked processes, stops kt's lane before go, and starts no lane after the failure. So a cheap failure ends the run before the Rust build finishes. The test also drops the runner's GITHUB_ACTIONS and GITHUB_STEP_SUMMARY, which would move the failure annotation to stdout and append to the real job summary. A second test sends SIGTERM to a run blocked in check rs, at 1 and 3 jobs, and asserts it exits 143, records check rs (cancelled), and leaves the blocked process dead.

In the queue run the quest cites, the bun lane (js, drafts, markdown, gh) sums to about 113 s and the rs lane to 240 s, so gh's failure would have ended the entry in about 2 minutes instead of 16.

Measurements

These were taken locally on a 24-core host shared with other agents, so the numbers are noisy. The 1-minute load average ran 10 to 70 during these runs. Each run appends a new comment to rs/moq-net/src/lib.rs and rs/moq-ffi/src/lib.rs, then runs MOQ_STRICT=1 NEXTEST_PROFILE=ci sh/dispatch.sh ci HEAD. That selects the rs dependents of moq-net, every binding, and the repository lints, like a multi-language queue entry. Caches were warm.

Run Before (serial, main) After, 3 lanes After, 4 lanes
a 331 s 282 s 204 s
b 294 s 182 s

Mean: 312 s before, 232 s after with 3 lanes (about 26% less). On the quieter pair (b) it is 38% less. 4 lanes was not clearly better than 3, so the default stays 3.

The win in the queue depends on rs and the ffi lane overlapping. Queue entries that select both spend 150 to 420 s in test rs and 400 to 500 s in the bindings, so I expect about 30% off them (870 s down to about 580 s). The quest estimated 45%, and dart running alone is most of the gap.

Public API and wire

No public API or wire change. MOQ_CHECK_JOBS is a new developer environment variable read by sh/dispatch.sh.

Checks

  • Merged main. Its carla check fails on main itself, since moq-dev/moq#6595 removed TrackProducer.flush but demo/carla/sim still calls it; moq-dev/moq#6639 fixes that, and this PR's CI needs it first. The local checks below ran with that fix applied.
  • python3 sh/dispatch-query.test.py (90 runs under GITHUB_ACTIONS=true, no failures), python3 sh/js/workspace.test.py, shellcheck, and shfmt pass.
  • quest check passes.
  • just check (every module, since dispatch changed) passes with nix flake check stubbed out, except check dart's load-sensitive leak test (see follow-ups), which then passed 2 of 3 standalone runs at load average 40. The flake stub is needed because this host's AppArmor apparmor_restrict_unprivileged_userns=1 refuses the unshare the runner image build needs (unshare: write failed /proc/self/uid_map), unrelated to this change. An unstubbed run hit that failure, and the cancel path ran for real: it cancelled the rs and ffi lanes mid-test and reported check flake failed.
  • The merge also drops links to this quest from obs-parallel.md (now unblocked) and python-setup.md, whose rebuild sample predates py's check and test sharing a lane.
  • Not run: just test interop --all (no wire, FFI, or gateway change).

Decisions

Settled with the maintainer:

  1. The knob stays MOQ_CHECK_JOBS, default 3, ignored by just fix.
  2. dart lands running alone. Giving the dart build hook its own Cargo build.build-dir, so dart can run beside other lanes, is a follow-up.

(Written by Claude Opus 5.5)

Follow-ups (suggested quests)

  • Isolate the dart hook's build dir, as decided above. Then dart can drop its run-alone rule.
  • The dart leak test is load sensitive. dart/moq_ffi/test/leak_test.dart's "a non-null optional argument is released" case fails about 20 to 40% of runs standalone on a loaded host: RSS grows 45 to 63 MB against a 32 MB bound. MALLOC_ARENA_MAX=1 does not help (4 of 10 failed). Each call throws a MoqException whose message echoes the 64 KiB reference, so the RSS likely includes Dart garbage the GC has not collected yet, not a native leak. It did not fail in the last 30 failed queue runs, since the runner was otherwise idle. Running dart alone keeps the queue there.
  • Watch the queue runner's peak memory on the first parallel entries (earlyoom kills at 10% free). Fold that into local-gate.md's measurements.

🤖 Generated with Claude Code

(Written by Claude Opus 5.5)

kixelated and others added 3 commits October 23, 2026 23:11
`just check` and `just ci` ran every selected module one after another.
They now run in lanes, MOQ_CHECK_JOBS (default 3) at a time, slowest
first. Modules that share mutable state stay serial in one lane: the
root Bun workspace (js, drafts, markdown, gh), and the venv, the
target/debug moq-ffi library, and cpp/moq/build (carla, py, kt, swift,
go, cpp, obs_compile). Concurrent lanes buffer their output and print it
when they finish; the first failure cancels the running lanes' process
groups, starts no more, and names its phase. fix stays serial.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dart builds moq-ffi with its build hook's filtered environment and
checks the hook then finds that build fresh. Any other Cargo build in
between reruns host build scripts that track the environment, such as
bindgen's BINDGEN_EXTRA_CLANG_ARGS, so the hook rebuilt and the check
failed whenever another lane compiled beside it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated kixelated changed the title Run the queue's modules in parallel ci: run check and ci modules in concurrent lanes Oct 24, 2026
@kixelated

Copy link
Copy Markdown
Collaborator Author

Outcome: the quest's work is done, and the PR stays a draft for two maintainer decisions, listed in the body: the MOQ_CHECK_JOBS name and default, and whether dart keeps running alone.

Measured on a loaded local host: a moq-net plus moq-ffi entry takes 312 s serial and 232 s with 3 lanes on average, the best pair going from 294 s to 182 s. Two races found while measuring are fixed by lane grouping, and a third by running dart alone. The dart leak test is load sensitive on its own; that is recorded as a follow-up.

(Written by Claude Opus 5.5)

kixelated and others added 2 commits October 24, 2026 07:40
# Conflicts:
#	quest/m0/ci/parallel-modules.md
…s env

The quest asks that an injected 'just gh test' failure end the run before
the Rust build finishes and leave no module running. Fail gh's tests once
rs's tests and kt's check are both blocked, so the run ends only if both
lanes are cancelled.

Drop the runner's GITHUB_ACTIONS and GITHUB_STEP_SUMMARY from the test's
environment: under Actions the failure annotation goes to stdout and the
report appends to the real job summary.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
obs-parallel is unblocked and takes its bound from MOQ_CHECK_JOBS.
python-setup's rebuild sample predates parallel modules, where py's test
now runs right after its check in the same lane.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@kixelated
kixelated marked this pull request as ready for review October 24, 2026 15:03
@coderabbitai

coderabbitai Bot commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 18 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 53a0ad49-52a3-4aef-b604-0328f2f63ba5
Commits

Reviewing files that changed from the base of the PR and between bf6d4aa and a364c6f.

Files selected for processing: 8
  • quest/m0/ci/README.md
  • quest/m0/ci/local-gate.md
  • quest/m0/ci/nightly-levers.md
  • quest/m0/ci/obs-parallel.md
  • quest/m0/ci/parallel-modules.md
  • quest/m0/ci/python-setup.md
  • sh/dispatch-query.test.py
  • sh/dispatch.sh

No actionable comments were generated in the recent review.

Recent review info
Run configuration
  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 7cc958a3-f8a2-4e9f-a56c-7069c91d3981
Commits

Reviewing files that changed from the base of the PR and between a3c633d and bf6d4aa.

Files selected for processing: 8
  • quest/m0/ci/README.md
  • quest/m0/ci/local-gate.md
  • quest/m0/ci/nightly-levers.md
  • quest/m0/ci/obs-parallel.md
  • quest/m0/ci/parallel-modules.md
  • quest/m0/ci/python-setup.md
  • sh/dispatch-query.test.py
  • sh/dispatch.sh
Files with no reviewable changes: 3
  • quest/m0/ci/README.md
  • quest/m0/ci/parallel-modules.md
  • quest/m0/ci/nightly-levers.md

Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 0 remain after this review.

Walkthrough

sh/dispatch.sh now runs check and CI module lanes with bounded concurrency, adds test phases for selected modules, reports phase timings, and cancels active lanes after failure or termination. The tests cover lane failure and dispatcher termination. Quest documents now describe dispatcher lane concurrency and update or remove references to the separate parallel-modules proposal.

Priority: Normal

Merge Risk: Minimal · up to bf6d4

No actionable issue remains from the reviewed changes; the PR is mergeable after normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 15 functions across 2 files. (3 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Title check Passed The title clearly and concisely describes the main change: running check and CI modules in concurrent lanes.
Description check Passed The description directly explains the concurrent lane scheduling, concurrency control, grouping rules, cancellation behavior, measurements, and validation for the changeset.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Full details: Docstring Coverage

Explanation

Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 15 functions across 2 files. (3 skipped: 3 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches
✨ Simplify code
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-24T16:20:51.604809Z a364c6f New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 4816395dcc

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread sh/dispatch.sh Resolved
A serial run (MOQ_CHECK_JOBS=1, and every 'just fix') left its lane in
the terminal's process group, so SIGTERM reached only the lane shell,
which defers its trap until the phase exits: the workload and the
dispatcher kept running. Give every lane its own process group.

A phase's subshell also died on TERM without reaping its children, so a
cancelled run could return while they were still exiting. Trap TERM
there too, so the subshell exits once its phase does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	quest/m0/ci/README.md
#	quest/m0/ci/nightly-levers.md
@kixelated

Copy link
Copy Markdown
Collaborator Author

Merge summary: merged main into the branch (a364c6f). The only conflicts were quest links: moq-dev/moq#6616 finished one-test-build.md and this PR finishes parallel-modules.md, so quest/m0/ci/README.md and nightly-levers.md drop both links. Main's sh/rs/select.sh change is independent of the lane logic.

moq-dev/moq#6639 is now on main, so the all-module dispatch's carla check has its fix. Locally sh/dispatch-query.test.py, sh/rs/select.test.py, and quest check pass. Codex reviewed a364c6f with no findings and CI is green. The maintainer settled both decisions in the body (MOQ_CHECK_JOBS default 3, dart runs alone). The follow-ups listed in the body are still open.

(Written by Claude Opus 5.5)

@kixelated
kixelated added this pull request to the merge queue Oct 24, 2026
Merged via the queue into main with commit fb1c0ca Oct 24, 2026
10 checks passed
@kixelated
kixelated deleted the quest/m0/ci/parallel-modules branch October 24, 2026 18:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant