Repository navigation
CI: merge queue serializes entries (~25 min each) #1130
Description
Problem
The main merge queue built and merged one entry at a time (max_entries_to_build: 1, max_entries_to_merge: 1). Each entry takes 13–25 minutes, set by the slowest required workflow: Shared Server fixtures (~23–25 min) or PHPUnit Feature (~13–17 min). Measured on the last 12 merge-group runs before 2026-11-11 16:55 UTC. Seven queued PRs therefore serialize to about 3 hours.
Change 1 (done 2026-11-11)
Ruleset "Server main merge queue" now builds and merges up to 5 entries together (max_entries_to_build: 5, max_entries_to_merge: 5, grouping ALLGREEN). Required status checks are unchanged. Each group still runs every required check on the combined commit. The repo is public, so hosted-runner minutes are not billed.
Next
- Measure queue time for the PRs currently queued (Answer 404 for an empty trailing path segment instead of 501 (#1006) #1122 Prune expired runs from the native maintenance loop (#633, #500) #1124-List promote in the control-plane build-id intent enum (#521) #1128) and record it here.
- If the fixtures job still dominates: path-filter or split it so Rust-only changes skip PHP-only suites, keeping required coverage. Then consider caching the fixture build.
Owner: server. Requested by General.
Activity
rmcdaniel commentedon Nov 11, 2026
Flake seen in a grouped queue run (43064230589, entry for #1125):
cross_kind_poll::remote_workflow_admission_interrupts_both_other_kinds_without_a_leasegotpoll_status: emptyinstead oftask_queue_changed(tests/execution/cross_kind_poll.rs:80); 763 other tests passed. Unrelated to #1125's change. Re-queued #1125. If it recurs under grouped runs, the test's wait window needs widening.rmcdaniel commentedon Nov 11, 2026
Measurement after grouping (2026-11-11): five PRs (#1122 #1124 #1126 #1127 #1138) built as one group and merged together at 17:50 UTC, about 55 minutes after the queue picked up the new setting. Serialized at 13-25 min each, the same five would have taken about 1.5-2 hours. Required checks unchanged. Two grouped-run flakes so far: cross_kind_poll (noted above) and an image-pull exit 125 on a PR run. Next: watch a few more groups; only path-filter or split the fixtures job if queue time is still the bottleneck.
rmcdaniel commentedon Nov 11, 2026
Another runner-side failure pattern:
docker runexits 125 when pulling a published image (Docker Hub auth token fetch on #1122; theghcr.io/durable-workflow/serverPHP 2.11.x upgrade image on #1124, #1137 and #1144cohortsjobs). Three today, each fixed by a rerun. If it continues, pre-pull those images with a retry inscripts/ci/pull-images.shbefore the profile steps.rmcdaniel commentedon Nov 12, 2026
Flake: #1153 PR run 43130557498, job
postgres-storage (existing-php-matrix, corrected-markers). Every php and rust comparison passed, then the embedded continue-as-new memo probe exited 100 with no output (01:05:48Z). That probe is unrelated to the PR's schedule change. Failed jobs rerun.rmcdaniel commentedon Nov 12, 2026
Measurement and next step (General).
Since grouping went live (28 PRs merged after 2026-11-11 17:00 UTC), enqueue → merged takes median 24.6 min, mean 30.6. The floor is set by the fixtures (~24 min) or by PHPUnit (~16 min) plus Sonar (~1 min). Runners are not a constraint: fixture jobs start within 1–6 s even with 5 groups building.
Redundant work. In 7 of 24 groups I could check, the group commit's tree was identical to the tree the PR's own passing run had tested: #1141 #1149 #1160 #1163 #1164 #1165 #1166, and #1173 since. Each still re-ran the suites (17.7–27.3 min). #1176 skips the suites for exactly those groups: same first parent, PR head and tree as a successful same-repo
pull_requestrun of the same workflow. Contracts, the action policy, Public Boundary and Sonar still run. Expected effect for those entries: ~4–5 min instead of ~18–27. Needs Security review before merge.Ruleset: grouping is already set (
max_entries_to_build: 5,ALLGREEN). One optional change, which I have not applied:"merge_queue": { "merge_method": "MERGE", "max_entries_to_build": 5, "min_entries_to_merge": 1, "max_entries_to_merge": 5, "min_entries_to_merge_wait_minutes": 0, "grouping_strategy": "ALLGREEN", - "check_response_timeout_minutes": 360 + "check_response_timeout_minutes": 60 }The longest required check measured is 33 min (fixtures), and PHPUnit's job timeout is 25 min. A check that never reports (for example a lost runner or a missing Sonar App status) currently holds the head of the queue for 6 h before eviction. 60 min still leaves about twice the worst observed run. The cost is that an evicted entry must be re-queued. I don't recommend
HEADGREEN: it would stop verifying non-head entries.The remaining entries are stacked or were tested on an older main. For those, the next lever is the fixture critical path: a 4.5 min native build, then a 16–19 min matrix.
rmcdaniel commentedon Nov 12, 2026
General (2026-11-12): two decisions.
check_response_timeout_minutesdrops from 360 to 90 (ruleset 25900633, the only change). The slowest real check measured was 33 min, so a check that never reports now blocks the queue head for at most 1.5 h instead of 6 h. Grouping staysALLGREENand does not move toHEADGREEN, so every entry stays verified.rmcdaniel commentedon Nov 12, 2026
General (2026-11-12): fixed by #1176 plus the ruleset timeout change, and confirmed in production. #1178's merge group (
…4ae3c006f750) ran from 21:41 to 21:43, about 2 min end to end (PHPUnit Feature, Shared Server fixtures and Public Boundary). Earlier groups took 21–26 min (for example, 19:53–20:19). Its tree matched its PR's tested run, so the heavy suites were skipped through the tested-merge marker. Groups that don't match still run everything. The queue's check timeout is now 90 min (was 360).CI: skip queue suites already passed on the exact same merge (durable…
Merge pull request durable-workflow#1176 from durable-workflow/ci/113…