Skip to content

ci: run leaf corpus gates once across the matrix, not per job - #23461

Closed
leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:claude/leaf-gates-sharded
Closed

leonlaiyc wants to merge 1 commit into
kirodotdev:mainfrom
leonlaiyc:claude/leaf-gates-sharded

Conversation

@leonlaiyc

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A test-only pull request's Backend Tests (3.12) jobs finish inside their 60-minute limit, because leaf mode runs each corpus gate once across the matrix instead of once on every job.

On a fork PR whose diff touches only test/ files, CI runs in leaf mode, and each of the eight Backend Tests (3.12, N) jobs runs the whole leaf corpus gates (692 file(s), once) group before the changed files. That group is now a full shard's worth of tests, so jobs on ubuntu-latest are cancelled at timeout-minutes: 60, often before they reach the PR's own tests (#23450).

Second occurrence, on #23389 (run 39926606277): four of eight jobs were cancelled. Shard 7's gate group reported 35469 passed ... in 3537.43s (0:58:57) and the job was cancelled 11 s into leaf changed files, pass 2/3. Every test that ran passed; the run went red only on time.

Why it matters

The leaf scope exists so a one-line flake fix gets CI quickly. Right now a test-only PR runs about 8 × 35k gate tests, takes longer than a full run, and goes red on job time with nothing actually failing. A fork author cannot rerun it (gh run rerun needs admin rights), so every such PR waits on a maintainer rerun that is likely to time out again. That is what keeps flake fixes, which unblock every other open PR, sitting red.

Not a goal

  • Changing which files scripts/leaf_test_scope.py derives as corpus gates. Over-including is the safe direction, and the derivation stays as it is.
  • Changing the job's timeout-minutes, the shard count, or the full / reduced / Windows test paths.
  • Sharding the changed-file repeats. Each job stays an independent roll of the changed files, three times over, which is what a flake fix needs.

What changed (motivation → approach → change)

  • Symptom: leaf-mode Backend Tests jobs hit the 60-minute limit, inside the corpus-gate group or just after it.
  • Root cause: the step comment says "the corpus gates run ONCE", but that happens once per job: all eight matrix jobs run the same full gate list. It was ~158 files at ~3 minutes when the reduction was designed. corpus_gates() now derives 692, which is about as many tests as one full shard (shard 7: 35,469 tests, 59 min). The other seven copies add nothing, because the gates are deterministic content assertions.
  • Change (.github/workflows/ci.yml, backend-test Run tests step, leaf branch only): before the gate group, each job keeps only the gates that scripts.ci_file_shards.file_shard assigns to its matrix.group, the same owner function the full suite uses. Across the matrix the gates still run exactly once, now about 1/8 of the group per job. A job that owns no gate prints a line and skips the group. A job that owns some still runs them through the normal pytest, so exit 5 (nothing collected) stays a failure there. I filtered in the shell instead of loading the plugin because a job that legitimately owns nothing would also get exit 5. The changed-file repeat loop is unchanged.
  • docs/ci/ci-and-reviews.md § Backend file sharding now says leaf corpus gates are partitioned across the matrix.

Backwards compatibility

Compatible: only the leaf branch of one CI step changes. Full, reduced, Windows, IPv6 and kernel-lock lanes are byte-identical, and the leaf union still runs every gate once.

Removes nothing: every corpus gate still runs on exactly one job of the matrix, and the changed files still run LEAF_REPEAT times on every job.

Tests

  • test/test_leaf_test_scope.py::test_the_leaf_corpus_gates_run_once_across_the_matrix_not_once_per_job runs the REAL Run tests step body, pulled out of ci.yml, under bash -eo pipefail once per shard 1..8, with a pytest stub that records argv. It asserts the union of the gate runs is exactly the 40-gate list, each gate on its file_shard owner, at most one gate run per job, and the changed file run 3 times on every job.
  • test/test_leaf_test_scope.py::test_a_shard_that_owns_no_gate_skips_the_group_and_still_runs_its_repeats: with one gate, only its owner runs it, and every job exits 0 and still runs the 3 repeats.
  • test/test_ci_file_shards.py::...test_workflow_only_changes_linux_and_windows_non_leaf_partitioning now pins that the gate part of the leaf branch uses file_shard and the repeat loop stays unsharded.
  • Negative control: with origin/main's ci.yml and the new tests in place, both new tests fail (shard 1 runs all 40 gates; the non-owner shard runs the single gate).
  • Every test file that reads ci.yml (58 files): 2,976 passed and 9 failed. All 9 fail the same way on a clean origin/main worktree on this Windows host (uv trampoline path, cp950 decoding of two YAML files, no symlink privilege, the PowerShell/Git Bash probe), so none comes from this diff.

Proof: not provable -- other: prove.py reports BASELINE_RED on this Windows host, because test_ci_file_shards.py already holds a host-local failure on clean main; the manual revert of ci.yml above is the negative control

Manual verification

N/A: CI itself is the integration surface. The step body that runs is the one in the workflow file, executed by the tests above under the same -eo pipefail flags that ci-shell uses. This PR is itself test-plus-workflow, so its own Backend Tests run is the first live leaf-mode check of the sharded gates. I can't trigger it in advance from a fork.

Related Issues

Fixes #23450

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

🤖 Generated with Claude Code

@leonlaiyc
leonlaiyc requested review from a team and KiroCrewCaptain as code owners October 23, 2026 12:42
@leonlaiyc
leonlaiyc requested a review from mbriones98 October 23, 2026 12:42
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 23, 2026
@leonlaiyc

Copy link
Copy Markdown
Contributor Author

Maintainers: this PR's checks are waiting on Approve and run, not failing. It's a fork PR that edits .github/workflows/ci.yml, so GitHub has held all 20 workflow runs for head ac8efbe05 as action_required, and none of them has run a job yet. PR Readiness lists them under Awaiting maintainer approval, which only a maintainer can clear.

The 60-minute cancellations this PR fixes (#23450) keep showing up on test-only PRs:

With this change, each shard runs only the gates that file_shard assigns to it, so a leaf run does the gate work once across the matrix. The step body is tested under bash in test/test_leaf_test_scope.py. Could someone approve the runs for ac8efbe05? Thanks.

In leaf mode every Backend Tests (3.12) job ran the whole corpus-gate
list before the changed files. The list is now ~700 files, a full
shard's worth of tests, so hosted jobs hit the 60-minute limit and a
test-only pull request went red on time with nothing failing.

Each job now keeps only the gates scripts.ci_file_shards.file_shard
assigns to its matrix group, so the union still runs every gate once.
A job that owns none skips the group; the changed-file repeats stay
whole on every job.

Fixes kirodotdev#23450

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@iamwhatever

Copy link
Copy Markdown
Contributor

Thank you, @leonlaiyc! Issue #23450 (fork leaf-corpus gate timeouts) was closed as done on main via #30115, so this is no longer needed. Closing; please reopen if the gates still run per job.

@iamwhatever iamwhatever closed this Nov 6, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CI: leaf-mode Backend Tests jobs run every corpus gate and hit the 60-minute limit on hosted runners

2 participants