Repository navigation
ci(obs): check OBS sources concurrently - #6684
Conversation
Quest-Claim: 2788d4f4d28e88d2
obs_compile still sits on the critical path after module lanes landed: it is the last module of the serial ffi lane that dart waits on. Check its 20 translation units under the MOQ_CHECK_JOBS bound once headers are ready, buffering each source's diagnostics and reaping every compiler's process group on cancel. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
No actionable comments were generated in the recent review.
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a7898b5e18
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
A compiler killed outright, such as by the OOM killer under concurrent checks, prints nothing that says which source it was checking. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bash 5.2 drops a background job that a signal ends from its job table once any other job is collected or a foreground command finishes, so wait -n reports "no such job" and the gate aborts on an unset pid, losing that source's diagnostics. Run each compiler as a child of a subshell job, which exits normally with the compiler's status. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
@codex review (Written by Claude Opus 5.5) |
|
Codex Review: Didn't find any major issues. Nice work! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
Merging 6ba8d94 with the maintainer's approval. The non-Grok review of this head is clean, every finding is fixed or answered, and the pre-land (Written by Claude Opus 5.5) |
just obs compilenow syntax-checks OBS's 20 source and test translation units concurrently, up toMOQ_CHECK_JOBS(default 3) at once, after header preparation finishes. Completes thequest/m0/ci/obs-parallelquest.Is OBS still on the critical path?
Yes. I checked this first, since moq-dev/moq#6617 landed module lanes.
ffilane took 515s (carla 4, py 265, kt 85, go 2, cpp 123, obs_compile 36), thendartran alone for 109s. Wall time was 624s.obs_compileis the last module in that lane, anddartwaits for it, so all 36s were on the critical path. Header prep was a no-op there (Cargo "Finished in 0.36s", every install "Up-to-date"), so nearly all of the 36s was compiling.cpp/moq/orrs/moq-ffi/runcppandobs_compileback to back in theffilane, so the same holds.cpp/obs/orsh/obs/: before this change, locallyobs_compile(about 21s) was the longest module. Next wereghtest (about 25s in CI) and shell (12s). Nowobs_compiletakes 9s andghtest (26s) finishes last.Only one queue entry has run
obs_compilesince module lanes landed, so the per-scenario numbers above combine it with local runs.Measurements
Local runs, 24 cores with load average around 10 from other work, same inputs, warm build:
includes.sh, warm)compile.sh,MOQ_CHECK_JOBS=1(serial)compile.sh,MOQ_CHECK_JOBS=3(default)Checking each translation unit serially takes 0.1s to 2.0s. The slowest are
moq-dock.cpp,moq-advanced-dialog.cpp, andobs-moq.cpp, which pull in Qt or all of libobs. A cold prep (building moq-ffi and the C++ package) took 48s once and isn't affected by this change.Verification
src/moq-service.cpp,test/moq-rate-test.cpp): both errors were reported whole, all 20 sources were checked (counted through a loggingCXXwrapper), and the command exited 1.CXXstub that forks a long-lived child (like GCC's driver forkingcc1plus). TERM tocompile.shalone, TERM to its process group (howsh/dispatch.shcancels a lane), and INT to its group (Ctrl-C) exited 143/143/130, with 3 children before and 0 after in each case.MOQ_CHECK_JOBS=8with a stub that fails two sources and SIGKILLs a third (an OOM kill): all 20 checked once each, every failure named, exit 1. HUP and INT tocompile.shalone also exited 129/130 with no compilers left and the log directory removed; a success leaves no log directory either.catorwait -n, sowait -nreportedno such joband the gate aborted onpid: unbound variable, losing that source's diagnostics. A stub that SIGKILLs every third source hit it in 20 of 25 runs on bash 5.2.15. With each compiler run as a child of a subshell job, it passed 25 of 25 on 5.2.15 and 10 of 10 on 5.3.20 (the dev shell's), every killed source named. Normal exits stay in the table in script mode (up to CHILD_MAX), so they were never affected.MOQ_CHECK_JOBS=0fails with the same error messagesh/dispatch.shgives.nix develop --command just checkpassed (obs_compile, obs, markdown, gh, quest, shell, large_files, toml, nix, justfile).Decisions made without asking
MOQ_CHECK_JOBSwith the same default of 3 and the same validation assh/dispatch.sh, rather thannprocor a new variable, so there's one knob. The "local admission share" the quest mentions doesn't exist yet. quest/m0/ci/local-gate.md owns it, and I noted this second consumer there.set -monly around the launch), and the EXIT trap sends TERM to each group, then waits. GCC's driver doesn't pass a signal on tocc1plus, so killing only the driver pid would leave the compiler running.( ...; exit )job rather than as the job itself, so the job exits normally even when the compiler is killed, which keeps its status collectable bywait -non bash 5.2. The trailingexitstops bash exec'ing the compiler in the subshell's place. The cost is one extra bash fork per source.error: <source> failed with status Nline, since a compiler killed outright (more likely with three Qt checks at once) prints nothing naming its source.sh/dispatch.shuntouched: I didn't add a cross-reference comment there. Any change to that file makes the dispatcher run every module, which would turn this PR's checks into a full 10-minute run.Impact
just obs compile(and theobs_compilecheck module) now runs up toMOQ_CHECK_JOBScompilers at once, and rejects a value that isn't a positive integer. Diagnostics go to stderr and print per source in completion order, not alphabetically.Suggested follow-ups
ghtest phase (about 25s) now endscpp/obs-only entries and runs on every diff. It may be worth a look if small entries matter.ffilane (515s) plusdartrunning alone (109s) is the whole critical path.py(265s) andcpp(123s) dominate it, so splitting that shared-state lane would save far more than this change.🤖 Generated with Claude Code
(Written by Claude Opus 5.5)