Repository navigation
ci: refuse a backend shard whose test-file list differs from the run's - #40392
Conversation
A --file-shard-targets list is placed across the shards by measured time, so every shard of a run must hold the same list: shards holding different lists can each leave a file to another, and that file runs nowhere on a green run. The plugin could refuse such a shard since #40348, but no workflow passed it a digest. The changes job, which runs once per run and is not repeated when one failed shard is re-run, now names the digest of each list: the leaf corpus gates, the reduced scope on Linux and on Windows, and the docs contract readers. Every pytest call that passes a list passes that digest with it. A shard whose list differs fails before collecting, as does one that holds a list the run named no digest for, and one that holds no list while the run has one. A full-suite run passes neither. ci-surface-tests.py --for-windows prints the Windows list on any host, so the Linux changes job names the Windows digest from the selector the Windows shards run. The plugin refuses a value that is not a digest. Closes #40363
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS No design-level concerns: the Notes
[DESIGN-REVIEWED] 1208a6e Description read[DESCRIPTION-READ] cea7f302dd2fdf5857cb5c18271b06b9dee8d24188bf47709ec05aad49acc271 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Every item comes from issue #40363 and the base plugin option that no workflow passed, and the change wires up the digest that already exists instead of adding a second mechanism. What this change shipsInventory (8 items) — 8 justifiedIntent: make a selected CI run go red when its backend shards hold different test-file lists, instead of going green while skipping files. FIX.
[FIRST-PRINCIPLES-REVIEWED] 1208a6e Description read[DESCRIPTION-READ] 43495c9d8a58579caab13097670173a07d3a72e62c3e97584416e342ef593b75 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
The changes job names the Windows shards' digest on Linux with ci-surface-tests.py --for-windows. Two things in the selector followed the host and could make a Windows shard's list differ from it: the glob, which ignores case on Windows and not on Linux, and the encoding of the printed list, which is the code page on Windows. The Windows list now matches test file names whatever their case by name, on both hosts, and the selector prints UTF-8. No tracked file is affected today: none differs from test_*.py by case only, and none has a non-ASCII name. A property test reads one synthetic tree as the Linux job does and as a Windows shard does (CRLF working tree, CRLF list file, a Windows-form root) and holds the two digests equal.
The POSIX list still left name matching to rglob, whose case rule follows the host: on Windows it ignores case. The Windows shards run test_ci_surface_tests.py, and the test that holds the Linux and Windows digests equal asks for the POSIX list there, so it would have failed on every Windows shard. Both lists now match names in the selector: case must match for the POSIX list, and is ignored for the Windows one. The lists of the real tree are unchanged on Linux, and a Windows run always takes the Windows list. The property test runs each seed under both glob rules.
|
GPT 6.1 finding on The finding is right. The POSIX list still left name matching to The class is "a list that follows the glob of the host it is computed on", so the fix is in the selector, not the assertion:
AI-assisted, posted by Claude Code on behalf of Zedmor |
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 868 lines, more than the 400 it may approve on its own. Each of the five sharded-list pytest calls in ci.yml now passes a digest that the changes job computes once per run, using the same scripts on the same merge commit. I checked that the selector uses only the standard library and reads no installed packages, that the Linux list is unchanged (still case-sensitive), and that the Windows list matches names ignoring case in the script itself, as the Windows glob did before, with the ignore filter kept. The shell sort/sha256 digest equals the plugin's for the generated lists, and the docs-readers diff and split are the same in the changes job and each docs leg. A shard whose list differs, that has a list the run named no digest for, or that has no list while the run has one is refused (fail closed). If the selector fails everywhere the full suite still runs as before. Re-running one shard keeps the same digest because changes is not re-run. The new tests run the workflow's own steps for all five paths and are deterministic (fixed seeds); all CI checks pass, Windows shards included. No blocking issues. Tier: pending: being reviewed. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved. Checks run on this head:
- All five
--file-shard-targetscall sites in ci.yml now pass the digest that thechangesjob computes. Its bash digest gives the same bytes as the plugin'stargets_digest. - The new test/test_ci_shard_list_digest.py passes 32 of 32. The same tests, plus the 13 new or edited selector tests, fail when ci.yml and the two scripts are reverted to the merge base.
- Six of eight mutations to the workflow, plugin and selector turn the suite red.
- Every Linux and Windows backend shard is green.
Done well: the test runs the workflow's own bash steps and feeds the result through the plugin's real pytest_configure. A behaviour change in ci.yml turns it red, and a wording change does not.
Optional:
- The digest path first runs end to end on the next reduced, leaf or docs-only PR. If a shard is refused there, compare its
file-shard targetssha256 line with theList digests:line in thechangesjob. - Removing
LC_ALL=Cfrom the digest sort, or removing the selector's UTF-8 stdout reconfigure, leaves every test green. Both fail closed. A non-C collation in theRun.stepenv would pin the first. - The Backwards compatibility line could also say that a shard whose selector fails while the run named a list now fails instead of running the full suite.
Problem / Motivation
Goal: A CI run whose backend shards were given different test-file lists fails on the shard that differs, instead of going green with tests that ran on no shard.
Since #40348,
scripts/ci_file_shards.pyplaces a--file-shard-targetslist across the shards by measured time, so the placement depends on the whole list. If two shards of one run hold different lists, each can leave a file to the other: that file is owned by no shard, its tests do not run, and the run is green. No shard can see another shard's list. #40348 added--file-shard-targets-sha256=<digest>to the plugin, but no workflow passed it. Issue #40363.Why it matters
A selected run (reduced scope, leaf corpus gates, docs contract readers) is trusted to have run every file on its list. Today that holds only because each shard happens to compute the same list from the same commit; nothing checks it, and a single shard that falls back to the full suite silently skips its share of the list. #39689 (a selected Windows test set) is blocked on this check.
Not a goal
What changed (motivation → approach → change)
Where each list comes from today, read from the code:
changesjob outputleaf_gatesbackend-test(env),backend-test-windows(file written by its scope step)ci-surface-tests.py --surface backendbackend-testtest/windows-collect-ignore.txtbackend-test-windowsgit diff HEAD^1 HEADthendocs_only.py readersdocs-contract-testsThe issue lists the docs readers as coming from one
changesoutput; they are computed per leg, so they get a digest too. No other workflow passes--file-shard-targets.Approach: the
changesjob runs once per run and is not repeated when one failed shard is re-run, so it is the one place that can name what every shard must hold.changesgets a step, "Digest the selected backend test lists", and four outputs:leaf_gates_sha256,reduced_sha256,reduced_windows_sha256,docs_readers_sha256. It works each list out the way the shards do, from the same commit, and publishesLC_ALL=C sort -u | sha256sumof it. Empty means the run has no such list. The outputs are 64 characters each, so output size limits do not come into it.scripts/ci-surface-tests.py --for-windowsprints, on any host, the list the selector prints on Windows, so the Linuxchangesjob can name the Windows digest from the same code the Windows shards run.test_*.pyby case only, none has a non-ASCII name).ci.ymlthat passes--file-shard-targets(five: reduced and leaf gates on Linux and Windows, docs readers) now also passes--file-shard-targets-sha256with the run's digest, fromenv:, quoted, with=.Unchanged: a full-suite run has no list, gets no digest, and its pins and hashes are as before. When the selector fails in
changesand on every shard, there is no list anywhere and the full suite runs, as before.push,pull_requestandmerge_grouptake the same path; no trigger, permission, action pin, timeout or gate is touched.Docs updated in the same commit:
docs/system-specs/common/testing-conventions.md(Shard balance) anddocs/ci/ci-and-reviews.md.Backwards compatibility
Compatible: the workflow only adds an argument where a list is already passed, and the plugin option stays optional. A digest that was accepted before (64 hex characters, any case, stray whitespace) is still accepted.
Removes nothing: the one stricter rule is on a plugin option that no caller passed before this PR.
Tests
test/test_ci_shard_list_digest.py(new, 32 tests) executes the workflow's own steps under bash: thechangesdigest step, then each shard's scope and test steps, for all five paths. Stand-in programs replace the selectors (they print the list the scenario gives that shard) and pytest (it records its arguments). The recorded arguments then go through the plugin's own option table andpytest_configurein the test process. The test resolves the steps'${{ }}expressions from what the earlier steps wrote; it names no step command, environment variable or output.targets_digestof the set for shuffled lists with repeats and blank lines; nothing is named for a list the run does not have.$(id)) is refused.test/test_ci_surface_tests.py:--for-windowson a POSIX host gives the list the selector gives on Windows, from the function and from the command line. A property test (6 seeds, each under a POSIX and a Windows glob rule) reads one synthetic tree twice: as the Linux job does (LF files,--for-windows, the byte-sorted digest) and as a Windows shard does (every file rewritten with CRLF, the selector's Windows branch, its output as a CRLF list file with a blank line and a repeat, the plugin's digest from the real root and from aD:\\root with backslash-joined paths). The tree carriesTest_*.py,TEST_*.PY, a non-ASCII name, a directory with a space and ignore-listed names; the digests are equal. It fails when either list leaves the case rule to the host glob.Each of nine mutations of
ci.yml(drop the digest from one call, pass the Linux digest to Windows, do not refuse a missing list, do not sort, compute the Windows digest from the Linux list) turns the new file red.The new test file reads the workflow tree, so it joins the guard tier:
scripts/ci_guard_tier_membership.pyreportstest/test_ci_shard_list_digest.py | joined, and no file left.Manual verification
On the real tree, the shell digest the
changesstep computes equals the plugin's digest of the same file, for the Linux reduced list (1,382 files), the Windows one (1,361) and the docs readers ofREADME.md(1,730).Not verified until a real run: this PR changes backend files, so its own CI is a full-suite run. The new step runs in
changesand publishes four empty digests; no job passes a list. The first frontend-only, leaf-only or docs-only pull request after merge is the first run that passes a digest. Path form, line endings, sort order, case and encoding are shown equal by code and by the property test above. What only a Windows runner can show is that its working tree holds the same files as the Linux one at scope time and that each is readable there. If it does not, the Windows shards of that run fail with the plugin's message; they do not pass wrongly.Real-machine verification
Machine: own computer, Amazon Linux 2023, x86_64, 32 vCPU / 123 GiB
Before (main 3c975bd):
python -m pytest -n 4 -q --no-cov test/test_ci_shard_list_digest.pywithci.yml, the plugin and the selector as on main -> 32 failed (no digest step, no digest passed)After (head b916153): the same command -> 32 passed
Cleanup: own computer
Related Issues
Closes #40363. Unblocks #39689. Follow-up to #40348 (#40160); part of #39690.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commitsContribution License Agreement
Contributed under the project's Apache-2.0 license.
AI-assisted, opened by Claude Code on behalf of Zedmor