Skip to content

ci: refuse a backend shard whose test-file list differs from the run's - #40392

Merged
Zedmor merged 4 commits into
mainfrom
ci/refuse-differing-shard-list-40363
Nov 21, 2026
Merged

Zedmor merged 4 commits into
mainfrom
ci/refuse-differing-shard-list-40363

Conversation

@Zedmor

@Zedmor Zedmor commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A CI run whose backend shards were given different test-file lists fails on the shard that differs, instead of going green with tests that ran on no shard.

Since #40348, scripts/ci_file_shards.py places a --file-shard-targets list across the shards by measured time, so the placement depends on the whole list. If two shards of one run hold different lists, each can leave a file to the other: that file is owned by no shard, its tests do not run, and the run is green. No shard can see another shard's list. #40348 added --file-shard-targets-sha256=<digest> to the plugin, but no workflow passed it. Issue #40363.

Why it matters

A selected run (reduced scope, leaf corpus gates, docs contract readers) is trusted to have run every file on its list. Today that holds only because each shard happens to compute the same list from the same commit; nothing checks it, and a single shard that falls back to the full suite silently skips its share of the list. #39689 (a selected Windows test set) is blocked on this check.

Not a goal

  • Passing the list itself from one job to the shards. Each shard still works its list out; the digest holds the copies to one value.
  • Changing which files any scope selects, the shard counts, pins, weights, timeouts or any gate.
  • The Windows leaf budget in "Select test scope" (too many or too heavy changed files send a Windows shard to the full suite). Every Windows shard decides it from the same run-wide outputs and the checkout; it is not digest-checked here.
  • Turning on Windows test selection (ci: move the Windows backend shards to diff-based test selection after the Linux gate #39689).

What changed (motivation → approach → change)

Where each list comes from today, read from the code:

List Produced Reaches pytest in
Leaf corpus gates once, changes job output leaf_gates backend-test (env), backend-test-windows (file written by its scope step)
Reduced scope, Linux on each shard: ci-surface-tests.py --surface backend backend-test
Reduced scope, Windows on each shard: the same selector, which on Windows drops test/windows-collect-ignore.txt backend-test-windows
Docs contract readers on each leg: git diff HEAD^1 HEAD then docs_only.py readers docs-contract-tests

The issue lists the docs readers as coming from one changes output; they are computed per leg, so they get a digest too. No other workflow passes --file-shard-targets.

Approach: the changes job runs once per run and is not repeated when one failed shard is re-run, so it is the one place that can name what every shard must hold.

  • changes gets a step, "Digest the selected backend test lists", and four outputs: leaf_gates_sha256, reduced_sha256, reduced_windows_sha256, docs_readers_sha256. It works each list out the way the shards do, from the same commit, and publishes LC_ALL=C sort -u | sha256sum of it. Empty means the run has no such list. The outputs are 64 characters each, so output size limits do not come into it.
  • scripts/ci-surface-tests.py --for-windows prints, on any host, the list the selector prints on Windows, so the Linux changes job can name the Windows digest from the same code the Windows shards run.
  • The selector's Windows list no longer depends on the host that computes it. Two things followed the host: the glob, which ignores case on Windows and not on Linux, and the encoding of the printed list, which is the code page on Windows. Both lists now match test file names in the selector, never through the host glob (case ignored for the Windows list, required for the POSIX one), and the selector prints UTF-8. No tracked file is affected today (none differs from test_*.py by case only, none has a non-ASCII name).
  • Every pytest call in ci.yml that passes --file-shard-targets (five: reduced and leaf gates on Linux and Windows, docs readers) now also passes --file-shard-targets-sha256 with the run's digest, from env:, quoted, with =.
  • The plugin refuses a value that is not 64 hex characters, with its own message. An empty value (the run named no digest for a list this shard holds) is therefore a failure, not "no check".
  • A shard that holds no list while the run has one fails instead of running the full suite's share: the reduced-scope selector failing or printing nothing on a shard, a leaf shard with no gate, a docs leg with no backend reader. The other shards have already left that shard its share of the list, and the full suite's split would not run it.

Unchanged: a full-suite run has no list, gets no digest, and its pins and hashes are as before. When the selector fails in changes and on every shard, there is no list anywhere and the full suite runs, as before. push, pull_request and merge_group take the same path; no trigger, permission, action pin, timeout or gate is touched.

Docs updated in the same commit: docs/system-specs/common/testing-conventions.md (Shard balance) and docs/ci/ci-and-reviews.md.

Backwards compatibility

Compatible: the workflow only adds an argument where a list is already passed, and the plugin option stays optional. A digest that was accepted before (64 hex characters, any case, stray whitespace) is still accepted.

Removes nothing: the one stricter rule is on a plugin option that no caller passed before this PR.

Tests

test/test_ci_shard_list_digest.py (new, 32 tests) executes the workflow's own steps under bash: the changes digest step, then each shard's scope and test steps, for all five paths. Stand-in programs replace the selectors (they print the list the scenario gives that shard) and pytest (it records its arguments). The recorded arguments then go through the plugin's own option table and pytest_configure in the test process. The test resolves the steps' ${{ }} expressions from what the earlier steps wrote; it names no step command, environment variable or output.

  • Identical lists, in a different order on each shard: every shard is admitted and the shards own each listed file exactly once (all 8, 8 and 3 shards).
  • One shard with a file fewer, a file more or a file swapped: that shard is refused ("different list from the rest of the run"), the others are admitted.
  • The run named no digest and a shard holds a list: refused.
  • A shard holds no list and the run named one: the step fails, and no pytest call runs in its place.
  • Full suite: no list, no digest, pins as before. Selector failing everywhere: full suite.
  • The digests equal targets_digest of the set for shuffled lists with repeats and blank lines; nothing is named for a list the run does not have.
  • A value that is not a digest (empty, 63 or 65 characters, non-hex, a prefix, an embedded newline and option, $(id)) is refused.

test/test_ci_surface_tests.py: --for-windows on a POSIX host gives the list the selector gives on Windows, from the function and from the command line. A property test (6 seeds, each under a POSIX and a Windows glob rule) reads one synthetic tree twice: as the Linux job does (LF files, --for-windows, the byte-sorted digest) and as a Windows shard does (every file rewritten with CRLF, the selector's Windows branch, its output as a CRLF list file with a blank line and a repeat, the plugin's digest from the real root and from a D:\\ root with backslash-joined paths). The tree carries Test_*.py, TEST_*.PY, a non-ASCII name, a directory with a space and ignore-listed names; the digests are equal. It fails when either list leaves the case rule to the host glob.

Each of nine mutations of ci.yml (drop the digest from one call, pass the Linux digest to Windows, do not refuse a missing list, do not sort, compute the Windows digest from the Linux list) turns the new file red.

The new test file reads the workflow tree, so it joins the guard tier: scripts/ci_guard_tier_membership.py reports test/test_ci_shard_list_digest.py | joined, and no file left.

Manual verification

On the real tree, the shell digest the changes step computes equals the plugin's digest of the same file, for the Linux reduced list (1,382 files), the Windows one (1,361) and the docs readers of README.md (1,730).

Not verified until a real run: this PR changes backend files, so its own CI is a full-suite run. The new step runs in changes and publishes four empty digests; no job passes a list. The first frontend-only, leaf-only or docs-only pull request after merge is the first run that passes a digest. Path form, line endings, sort order, case and encoding are shown equal by code and by the property test above. What only a Windows runner can show is that its working tree holds the same files as the Linux one at scope time and that each is readable there. If it does not, the Windows shards of that run fail with the plugin's message; they do not pass wrongly.

Real-machine verification

Machine: own computer, Amazon Linux 2023, x86_64, 32 vCPU / 123 GiB
Before (main 3c975bd): python -m pytest -n 4 -q --no-cov test/test_ci_shard_list_digest.py with ci.yml, the plugin and the selector as on main -> 32 failed (no digest step, no digest passed)
After (head b916153): the same command -> 32 passed
Cleanup: own computer

Related Issues

Closes #40363. Unblocks #39689. Follow-up to #40348 (#40160); part of #39690.

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

AI-assisted, opened by Claude Code on behalf of Zedmor

A --file-shard-targets list is placed across the shards by measured
time, so every shard of a run must hold the same list: shards holding
different lists can each leave a file to another, and that file runs
nowhere on a green run. The plugin could refuse such a shard since
#40348, but no workflow passed it a digest.

The changes job, which runs once per run and is not repeated when one
failed shard is re-run, now names the digest of each list: the leaf
corpus gates, the reduced scope on Linux and on Windows, and the docs
contract readers. Every pytest call that passes a list passes that
digest with it. A shard whose list differs fails before collecting, as
does one that holds a list the run named no digest for, and one that
holds no list while the run has one. A full-suite run passes neither.

ci-surface-tests.py --for-windows prints the Windows list on any host,
so the Linux changes job names the Windows digest from the selector the
Windows shards run. The plugin refuses a value that is not a digest.

Closes #40363
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 1208a6e7da678c39afe3dbb6dba834cd788938e3 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 1208a6e7da678c39afe3dbb6dba834cd788938e3 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design-level concerns: the changes job runs once per run, so it is the right single place to publish each list's digest. Failing a shard that holds no list closes the case where one shard silently falls back to the full suite.

Notes

  • The digest is only as reliable as the changes job and the shards computing their lists the same way. Two copies of the logic exist: the inline digest() in bash and the plugin's targets_digest. The workflow-executing test pins them together; keep the docs-readers split (grep -v '^website/') in one helper so it doesn't drift from the docs leg's sed split.

[DESIGN-REVIEWED] 1208a6e

Description read

[DESCRIPTION-READ] cea7f302dd2fdf5857cb5c18271b06b9dee8d24188bf47709ec05aad49acc271

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 1208a6e7da678c39afe3dbb6dba834cd788938e3 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Every item comes from issue #40363 and the base plugin option that no workflow passed, and the change wires up the digest that already exists instead of adding a second mechanism.

What this change ships

Inventory (8 items) — 8 justified

Intent: make a selected CI run go red when its backend shards hold different test-file lists, instead of going green while skipping files. FIX.

  1. A shard whose list differs from the run's digest fails before collecting — justified
  2. The changes job publishes four per-run list digests — justified
  3. ci-surface-tests.py --for-windows prints the Windows list on any host — justified
  4. The selector matches file names itself, not through the host glob (case ignored for the Windows list) — justified
  5. The selector always prints UTF-8 — justified
  6. The plugin refuses a digest value that is empty or not 64 hex characters — justified
  7. A shard whose selector fails or finds nothing fails instead of falling back to the full suite, when the run has a list — justified
  8. A docs-contract leg with no backend readers fails when the run named a list — justified

[FIRST-PRINCIPLES-REVIEWED] 1208a6e

Description read

[DESCRIPTION-READ] 43495c9d8a58579caab13097670173a07d3a72e62c3e97584416e342ef593b75

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 1208a6e7da678c39afe3dbb6dba834cd788938e3 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1208a6e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 1208a6e7da678c39afe3dbb6dba834cd788938e3 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 1208a6e7da678c39afe3dbb6dba834cd788938e3 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1208a6e

Verdict parsed from the review's SHA-scoped output markers for commit 1208a6e7da678c39afe3dbb6dba834cd788938e3.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 1208a6e7da678c39afe3dbb6dba834cd788938e3 false-positive: <reason, 20 to 500 characters>

Akim Akimov added 2 commits November 21, 2026 10:30
The changes job names the Windows shards' digest on Linux with
ci-surface-tests.py --for-windows. Two things in the selector followed
the host and could make a Windows shard's list differ from it: the
glob, which ignores case on Windows and not on Linux, and the encoding
of the printed list, which is the code page on Windows.

The Windows list now matches test file names whatever their case by
name, on both hosts, and the selector prints UTF-8. No tracked file is
affected today: none differs from test_*.py by case only, and none has
a non-ASCII name.

A property test reads one synthetic tree as the Linux job does and as a
Windows shard does (CRLF working tree, CRLF list file, a Windows-form
root) and holds the two digests equal.
The POSIX list still left name matching to rglob, whose case rule
follows the host: on Windows it ignores case. The Windows shards run
test_ci_surface_tests.py, and the test that holds the Linux and Windows
digests equal asks for the POSIX list there, so it would have failed on
every Windows shard.

Both lists now match names in the selector: case must match for the
POSIX list, and is ignored for the Windows one. The lists of the real
tree are unchanged on Linux, and a Windows run always takes the Windows
list.

The property test runs each seed under both glob rules.
@Zedmor

Zedmor commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

GPT 6.1 finding on 09fc276861 (test/test_ci_surface_tests.py:525, POSIX-case assertion fails on Windows): upheld, fixed in a third commit.

The finding is right. The POSIX list still left name matching to rglob, which ignores case on a Windows host, and the Windows shards run that test.

The class is "a list that follows the glob of the host it is computed on", so the fix is in the selector, not the assertion: _iter_files now matches names itself for both lists (case must match for the POSIX list, is ignored for the Windows one) and never hands a name pattern to rglob.

  • Test: the property test now runs each seed under both glob rules, with Path.rglob made to ignore case as on Windows. Without the fix 6 of 12 fail; with it 12 pass.
  • No workflow behaviour changes: on the real tree the Linux and --for-windows list digests are the same as at 09fc276861, and a Windows run always takes the Windows list.
  • This is the branch's third commit, accepted by the maintainer session for this fix; no force-push.

AI-assisted, posted by Claude Code on behalf of Zedmor

@Zedmor
Zedmor enabled auto-merge (squash) November 21, 2026 10:58
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 21, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 868 lines, more than the 400 it may approve on its own.

Each of the five sharded-list pytest calls in ci.yml now passes a digest that the changes job computes once per run, using the same scripts on the same merge commit. I checked that the selector uses only the standard library and reads no installed packages, that the Linux list is unchanged (still case-sensitive), and that the Windows list matches names ignoring case in the script itself, as the Windows glob did before, with the ignore filter kept. The shell sort/sha256 digest equals the plugin's for the generated lists, and the docs-readers diff and split are the same in the changes job and each docs leg. A shard whose list differs, that has a list the run named no digest for, or that has no list while the run has one is refused (fail closed). If the selector fails everywhere the full suite still runs as before. Re-running one shard keeps the same digest because changes is not re-run. The new tests run the workflow's own steps for all five paths and are deterministic (fixed seeds); all CI checks pass, Windows shards included. No blocking issues.

Tier: pending: being reviewed.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 1208a6e7da · 868 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. Checks run on this head:

  • All five --file-shard-targets call sites in ci.yml now pass the digest that the changes job computes. Its bash digest gives the same bytes as the plugin's targets_digest.
  • The new test/test_ci_shard_list_digest.py passes 32 of 32. The same tests, plus the 13 new or edited selector tests, fail when ci.yml and the two scripts are reverted to the merge base.
  • Six of eight mutations to the workflow, plugin and selector turn the suite red.
  • Every Linux and Windows backend shard is green.

Done well: the test runs the workflow's own bash steps and feeds the result through the plugin's real pytest_configure. A behaviour change in ci.yml turns it red, and a wording change does not.

Optional:

  • The digest path first runs end to end on the next reduced, leaf or docs-only PR. If a shard is refused there, compare its file-shard targets sha256 line with the List digests: line in the changes job.
  • Removing LC_ALL=C from the digest sort, or removing the selector's UTF-8 stdout reconfigure, leaves every test green. Both fail closed. A non-C collation in the Run.step env would pin the first.
  • The Backwards compatibility line could also say that a shard whose selector fails while the run named a list now fails instead of running the full suite.

@Zedmor
Zedmor merged commit 2a2e9b5 into main Nov 21, 2026
101 checks passed
@Zedmor
Zedmor deleted the ci/refuse-differing-shard-list-40363 branch November 21, 2026 12:00
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: refuse a backend shard whose selected test-file list differs from the run's

2 participants