Repository navigation
test(ci): make shard-digest leaf sampling ignore pinned files - #40565
Conversation
The Windows leaf gate runs the full suite when a diff changes more than WINDOWS_LEAF_MAX_PINNED pinned files. The digest tests sample their changed files from the first 400 test files by name, so a PR adding test files that sort early could shift the sample onto pinned files and turn test_a_shard_holding_no_list_fails_when_the_run_has_one[windows leaf gates] red. Sample only from files the Windows pin file does not list, and add a test that adds leading files and expects the same result.
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is a test-only change that removes the order-dependence where it starts, in the sampled corpus. Notes
[DESIGN-REVIEWED] b03d565 Description read[DESCRIPTION-READ] 43e278659f924de9703dc3e81349f70df1bb1cbf710a67ef7f15eafe1051e251 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: ApproveWhy: the decider estimates a 5% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes. Test-only change: the corpus fixture now excludes Windows-pinned files (parsed with ci_file_shards.read_pins, shard count 8 from the backend-test-windows job env), so a sample can no longer exceed WINDOWS_LEAF_MAX_PINNED and push the windows leaf-gates case onto the full-suite fallback. The workflow's targets-list paths never apply pins (ci.yml:2296), so excluding pinned files loses no coverage. The fixture scopes match, the unpinned pool is still well over 400 files, and the new parametrised regression test reaches the same empty-gates ::error:: path. CI is green. Tier: 1, every tier gate passed. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved automatically by the code change approver, not by a person. The decider estimates a 5% chance that this change introduces a defect, within the 12% limit for approving on its own, and the change has at most 400 changed lines.
Test-only change: the corpus fixture now excludes Windows-pinned files (parsed with ci_file_shards.read_pins, shard count 8 from the backend-test-windows job env), so a sample can no longer exceed WINDOWS_LEAF_MAX_PINNED and push the windows leaf-gates case onto the full-suite fallback. The workflow's targets-list paths never apply pins (ci.yml:2296), so excluding pinned files loses no coverage. The fixture scopes match, the unpinned pool is still well over 400 files, and the new parametrised regression test reaches the same empty-gates ::error:: path. CI is green.
Tier: 1, every tier gate passed.
Criteria: no merge conflict, no owner block, no change request, the default branch, no security-sensitive path, no design document to agree on, the readiness status, the review lanes, the P0 history, the linked issues' tier, the review lessons, no concern label, the file count, a safe change category, not its own change, the decider, the security checklist, the change matches its title, no competing change.
Decider v3c (217fe607). To overrule it, request changes or dismiss this review.
Problem / Motivation
Goal: A PR that adds test files no longer turns
test_ci_shard_list_digest.pyred for a reason unrelated to its own change.test/test_ci_shard_list_digest.py::test_a_shard_holding_no_list_fails_when_the_run_has_one[windows leaf gates](added in #40392) does not read a real diff. It takes the first 400 files oftest/test_*.pyby name and samples 12 of them with a fixed seed as the "changed files". When a PR adds test files that sort near the front (for example threetest/test_acp_*.pyfiles), the window shifts and the sample changes. If the new sample holds more thanWINDOWS_LEAF_MAX_PINNED=4files listed inscripts/ci_file_shard_pins_windows.txt, the Windows scope step correctly falls back to the full suite and exits 0, and the test assertingcode != 0fails. On current main, adding three empty filestest/test_acp_a1.py..a3.pymoves the sample from 2 pinned files to 6.Why it matters
Any contributor whose PR adds test files can get a red Backend Tests shard from a test that has nothing to do with their change, and the failure message (a Windows budget notice) points them at the CI script instead of at the sampling. This already happened on an open PR.
Not a goal
WINDOWS_LEAF_MAX_PINNED, or either pin file.What changed (motivation -> approach -> change)
Symptom: the windows leaf-gates case fails when early-sorting test files are added. Root cause: the sampled corpus includes Windows-pinned files, so the pinned count in the sample (and with it the scope step outcome) depends on which files sort first in
test/. Change: thecorpusfixture now goes through a small_corpus()helper that drops every path listed inscripts/ci_file_shard_pins_windows.txt(parsed with the plugin ownci_file_shards.read_pins, shard count read from the Windows jobSHARD_COUNT) before taking the first 400. Every sampling test in the file uses that fixture, so all of them are covered by the one change; none of them needs a pinned file.Backwards compatibility
Compatible: test-only change; no runtime or workflow code is touched.
Removes nothing: only the test corpus fixture is narrowed, and no reader outside this test file uses it.
Tests
test_new_test_files_sorting_first_leave_the_windows_leaf_sample_failing[0..5]: adds 0 to 5 file names that sort before every real test file, rebuilds the corpus through_corpus(), and runs the windows leaf-gates scenario oftest_a_shard_holding_no_list_fails_when_the_run_has_one. With the pin filter removed from_corpus(), case[3]fails withleaf diff outside the Windows budget (13 ... changed file(s), 6 pinned; need 1-40 and <= 4); with it, all pass.Proof: not provable -- test-only change
Determinism: the whole file (38 cases) passed 3 times in a row serially, and once more in a seeded shuffled order of all 38 node ids.
Manual verification
N/A -- the behavior is the test outcome itself, covered above.
Real-machine verification
Machine: own computer, Amazon Linux 2023, x86_64, 64 vCPU / 123 GiB
Before (main 378ec94): add empty
test/test_acp_a1.py,test/test_acp_a2.py,test/test_acp_a3.py, thenpytest -n 0 test/test_ci_shard_list_digest.py -k no_list_fails->1 failed, 4 passed([windows leaf gates]: 6 pinned; need <= 4)After (head b03d565): same three files,
pytest -n 0 test/test_ci_shard_list_digest.py->38 passedCleanup: own computer; the three probe files deleted
Related Issues
Found while a PR adding three
test/test_acp_*.pyfiles went red on this case; follow-up to #40392.Sibling sweep
Swept: git grep -n "random.Random" -- test/test_ci_shard_list_digest.py
Sibling: test/test_ci_shard_list_digest.py -- changed here (all sampling sites read the one
corpusfixture)Pattern harvest
Not generalizable: the hazard is specific to a test that samples real repo files and feeds them to a step with a count budget over a second file list.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commitsContribution License Agreement
Contributed under the project Apache-2.0 license.