Repository navigation
Conversation
|
Maintainers: this PR's checks are waiting on Approve and run, not failing. It's a fork PR that edits The 60-minute cancellations this PR fixes (#23450) keep showing up on test-only PRs:
With this change, each shard runs only the gates that |
In leaf mode every Backend Tests (3.12) job ran the whole corpus-gate list before the changed files. The list is now ~700 files, a full shard's worth of tests, so hosted jobs hit the 60-minute limit and a test-only pull request went red on time with nothing failing. Each job now keeps only the gates scripts.ci_file_shards.file_shard assigns to its matrix group, so the union still runs every gate once. A job that owns none skips the group; the changed-file repeats stay whole on every job. Fixes kirodotdev#23450 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ac8efbe to
d1b0bc3
Compare
|
Thank you, @leonlaiyc! Issue #23450 (fork leaf-corpus gate timeouts) was closed as done on main via #30115, so this is no longer needed. Closing; please reopen if the gates still run per job. |
Problem / Motivation
Goal: A test-only pull request's Backend Tests (3.12) jobs finish inside their 60-minute limit, because leaf mode runs each corpus gate once across the matrix instead of once on every job.
On a fork PR whose diff touches only
test/files, CI runs in leaf mode, and each of the eightBackend Tests (3.12, N)jobs runs the wholeleaf corpus gates (692 file(s), once)group before the changed files. That group is now a full shard's worth of tests, so jobs onubuntu-latestare cancelled attimeout-minutes: 60, often before they reach the PR's own tests (#23450).Second occurrence, on #23389 (run 39926606277): four of eight jobs were cancelled. Shard 7's gate group reported
35469 passed ... in 3537.43s (0:58:57)and the job was cancelled 11 s intoleaf changed files, pass 2/3. Every test that ran passed; the run went red only on time.Why it matters
The leaf scope exists so a one-line flake fix gets CI quickly. Right now a test-only PR runs about 8 × 35k gate tests, takes longer than a full run, and goes red on job time with nothing actually failing. A fork author cannot rerun it (
gh run rerunneeds admin rights), so every such PR waits on a maintainer rerun that is likely to time out again. That is what keeps flake fixes, which unblock every other open PR, sitting red.Not a goal
scripts/leaf_test_scope.pyderives as corpus gates. Over-including is the safe direction, and the derivation stays as it is.timeout-minutes, the shard count, or the full / reduced / Windows test paths.What changed (motivation → approach → change)
corpus_gates()now derives 692, which is about as many tests as one full shard (shard 7: 35,469 tests, 59 min). The other seven copies add nothing, because the gates are deterministic content assertions..github/workflows/ci.yml, backend-testRun testsstep, leaf branch only): before the gate group, each job keeps only the gates thatscripts.ci_file_shards.file_shardassigns to itsmatrix.group, the same owner function the full suite uses. Across the matrix the gates still run exactly once, now about 1/8 of the group per job. A job that owns no gate prints a line and skips the group. A job that owns some still runs them through the normalpytest, so exit 5 (nothing collected) stays a failure there. I filtered in the shell instead of loading the plugin because a job that legitimately owns nothing would also get exit 5. The changed-file repeat loop is unchanged.docs/ci/ci-and-reviews.md§ Backend file sharding now says leaf corpus gates are partitioned across the matrix.Backwards compatibility
Compatible: only the leaf branch of one CI step changes. Full, reduced, Windows, IPv6 and kernel-lock lanes are byte-identical, and the leaf union still runs every gate once.
Removes nothing: every corpus gate still runs on exactly one job of the matrix, and the changed files still run
LEAF_REPEATtimes on every job.Tests
test/test_leaf_test_scope.py::test_the_leaf_corpus_gates_run_once_across_the_matrix_not_once_per_jobruns the REALRun testsstep body, pulled out ofci.yml, underbash -eo pipefailonce per shard 1..8, with apyteststub that records argv. It asserts the union of the gate runs is exactly the 40-gate list, each gate on itsfile_shardowner, at most one gate run per job, and the changed file run 3 times on every job.test/test_leaf_test_scope.py::test_a_shard_that_owns_no_gate_skips_the_group_and_still_runs_its_repeats: with one gate, only its owner runs it, and every job exits 0 and still runs the 3 repeats.test/test_ci_file_shards.py::...test_workflow_only_changes_linux_and_windows_non_leaf_partitioningnow pins that the gate part of the leaf branch usesfile_shardand the repeat loop stays unsharded.origin/main'sci.ymland the new tests in place, both new tests fail (shard 1 runs all 40 gates; the non-owner shard runs the single gate).ci.yml(58 files): 2,976 passed and 9 failed. All 9 fail the same way on a cleanorigin/mainworktree on this Windows host (uv trampoline path, cp950 decoding of two YAML files, no symlink privilege, the PowerShell/Git Bash probe), so none comes from this diff.Proof: not provable -- other: prove.py reports BASELINE_RED on this Windows host, because test_ci_file_shards.py already holds a host-local failure on clean main; the manual revert of ci.yml above is the negative control
Manual verification
N/A: CI itself is the integration surface. The step body that runs is the one in the workflow file, executed by the tests above under the same
-eo pipefailflags thatci-shelluses. This PR is itself test-plus-workflow, so its own Backend Tests run is the first live leaf-mode check of the sharded gates. I can't trigger it in advance from a fork.Related Issues
Fixes #23450
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
🤖 Generated with Claude Code