Repository navigation
ci: spread a selected test-file set across shards by measured time - #40348
Conversation
A --file-shard-targets list was owned by path hash, which evens out the whole suite but not a small part of it: two heavy files sharing a shard set the run's time while the other shards finish early. The plugin now places the listed files heaviest first onto the shard with the least time so far, from a per-platform weights file that scripts/ci_file_shard_pins.py writes next to the pin files. A listed file with no weight gets the median. Every shard computes the same placement from the list, the weights and the shard count. With no targets list nothing changes: the full suite keeps its pins and hashes, and the pin files are untouched. Closes #40160
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. The diff only touches CI scripts, weight data files, tests and docs. It adds or changes no user-visible control, so no screenshot is needed. [UX-REVIEWED] 9d58200 Description read[DESCRIPTION-READ] 5ae44f35ab88a9834b0f048be25ab4b59d6e87c9a27ed150eb518f1f95f32bd0 That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the placement only moves a listed file between shards, a stale weights file costs balance but never drops a test, and a revert puts back the hash behaviour exactly. Notes
[DESIGN-REVIEWED] 9d58200 Description read[DESCRIPTION-READ] 1d821500acdad100682d769b76598ebde8dd1db43aad442cff48d489971123f2 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: the spread-by-measured-time fix targets the real cause (hash ownership on a selected list) and keeps the full suite as it was. Two flags have no workflow consumer. Not checked: whether the main-red-watch and website hunks in the patch are this PR's or drift from its base. Notes
What this change shipsInventory (7 items) — 4 justifiedIntent: make a reduced-scope backend CI run end when its work is done, by spreading the selected test files across shards by measured time. FIX.
[FIRST-PRINCIPLES-REVIEWED] 9d58200 Description read[DESCRIPTION-READ] 919fea3395bfb8528e35e30e962ab92a2de683103c259d83378218ff87f7b38f That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
The three new target tests each started their own interpreter children, one of them with nested xdist workers, beside this module's other subprocess tests. They now check ownership in-process, and one test in its own xdist group starts a single child that collects every scenario. A placement holds only if every shard is given the same targets. Each shard now prints the count and SHA-256 of its target set in the terminal summary, and --file-shard-targets-sha256 refuses a list with another digest.
|
Pushed 04fbf52 in answer to the GPT 6.1 finding on 6203da3, plus one guard I owed. Finding (test/test_ci_file_shards.py:527, D8): upheld. The class is "a new test in this module that starts its own interpreter child". Three of this PR's tests were in it (the ones at lines 508, 533 and 553), about 23 children between them, three of those with nested
The subprocess tests that were in this file before the PR are unchanged; moving them is outside this PR. Guard for differing target lists. A placement holds only if every shard is given the same set of targets. A shard cannot see another shard's list, so nothing inside the plugin can refuse a mismatch without a value shared across the run, and sharing one needs a workflow change this PR may not make. What is in this push:
No workflow passes the option yet, so today a mismatch is visible, not refused. Closing it means computing the digest once per run and passing it to every shard; that is a follow-up under The PR body's sections from "What changed" down are updated to match; Goal, Why it matters and Not a goal are untouched. AI-assisted, posted by Claude Code on behalf of Zedmor |
For generated target lists drawn from both checked-in weights files, the files owned across all eight shard indices add up to the list: no gap and no file owned twice. Covers unmeasured targets, repeats in the input, pinned files as targets, a single target and more shards than targets.
|
Pushed 9d58200 as a third plain commit. AGENTS.md caps a PR at two commits and forbids force-push; the Crew Captain asked for this test before arming, so it is added on top instead of rewriting history. The PR is squash-merged. It adds one test, The fail-closed half of the list guard (the workflow passing AI-assisted, posted by Claude Code on behalf of Zedmor |
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 17187 lines, more than the 400 it may approve on its own. Selected test-file lists (--file-shard-targets) are now spread across shards heaviest-first by measured time, using platform weights files. Every shard and worker computes the same placement: it depends only on the set of files, the weights and the shard count, and ties break deterministically. With no targets list, ownership is unchanged from main (pin or hash). Every workflow list is built from the same commit or one changes-job output, so shards get identical sets. The pin generator writes weights only on a default run or when --weights-out names a file. Tests cover the placement properties, real collection and refusals. CI is green. Tier: 2, held for a person: reviewer-finding. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
This review checked head 9d58200. Every caller of file_shard and hash_shard passes no targets list, so the full suite keeps main's pins and hash ownership. The three jobs that pass --file-shard-targets build the same list on every shard. balance() gives the same answer on every shard and xdist worker, Linux and Windows. The 15 new or changed tests pass, and 11 of 13 single-line mutations of the plugin turn them red.
Well done: weights are whole milliseconds and every tie-break is spelled out, so the placement depends only on the set, and the property tests check the longest-first bound and that no file is skipped or owned twice across all 8 shard indices.
Optional:
test/test_ci_file_shards.py:1204still says the leaf and reduced runs keep pure hash ownership. They are now placed by measured time.DEFAULT_WEIGHTShas nomacoskey, but--platformchoices come fromDEFAULT_OUT. If #37422 lands too, a default macOS refresh hits a KeyError after writing the pin file. A test that assertsset(DEFAULT_WEIGHTS) >= set(DEFAULT_OUT)would catch it.- For #40363: if some shards fall back to the full suite and the others stay reduced, a target can go unrun, and a digest check alone will not catch it. Making the reduced or full choice once per run would close that.
#40392) * ci: refuse a backend shard whose test-file list differs from the run's A --file-shard-targets list is placed across the shards by measured time, so every shard of a run must hold the same list: shards holding different lists can each leave a file to another, and that file runs nowhere on a green run. The plugin could refuse such a shard since #40348, but no workflow passed it a digest. The changes job, which runs once per run and is not repeated when one failed shard is re-run, now names the digest of each list: the leaf corpus gates, the reduced scope on Linux and on Windows, and the docs contract readers. Every pytest call that passes a list passes that digest with it. A shard whose list differs fails before collecting, as does one that holds a list the run named no digest for, and one that holds no list while the run has one. A full-suite run passes neither. ci-surface-tests.py --for-windows prints the Windows list on any host, so the Linux changes job names the Windows digest from the selector the Windows shards run. The plugin refuses a value that is not a digest. Closes #40363 * ci: make the selector's Windows list the same on every host The changes job names the Windows shards' digest on Linux with ci-surface-tests.py --for-windows. Two things in the selector followed the host and could make a Windows shard's list differ from it: the glob, which ignores case on Windows and not on Linux, and the encoding of the printed list, which is the code page on Windows. The Windows list now matches test file names whatever their case by name, on both hosts, and the selector prints UTF-8. No tracked file is affected today: none differs from test_*.py by case only, and none has a non-ASCII name. A property test reads one synthetic tree as the Linux job does and as a Windows shard does (CRLF working tree, CRLF list file, a Windows-form root) and holds the two digests equal. * ci: match the selector's test file names by name on every list The POSIX list still left name matching to rglob, whose case rule follows the host: on Windows it ignores case. The Windows shards run test_ci_surface_tests.py, and the test that holds the Linux and Windows digests equal asks for the POSIX list there, so it would have failed on every Windows shard. Both lists now match names in the selector: case must match for the POSIX list, and is ignored for the Windows one. The lists of the real tree are unchanged on Linux, and a Windows run always takes the Windows list. The property test runs each seed under both glob rules. --------- Co-authored-by: Akim Akimov <zedmor@amazon.com>
Problem / Motivation
Goal: A reduced-scope backend run finishes when its work is done, because the selected test files are spread across the shards by measured time and no shard is left holding the heavy ones.
scripts/ci_file_shards.pyowns a--file-shard-targetslist by path hash. That evens out the full suite, where the pin files correct it, but not a small selected set: two heavy files that hash to one shard set the run's time while the other shards finish early. On the Windows lists CI runs today the slowest shard carries 1.37x to 1.50x the mean test time (table below).Why it matters
Diff-based test selection (#39690) is modelled to cut the slowest Windows shard from 22.7 to 7.7 minutes. That model assumes the selected files are spread by time. Without this the saving stops at whatever the unluckiest hash shard costs, and #39689 cannot turn on a selected Windows set and expect the modelled wall time.
Not a goal
--file-shard-targetsthe pins and hashes are as they are on main, and the two pin files are untouched..github/. The workflows already pass--file-shard-targets; the Windows leaf caps (WINDOWS_LEAF_MAX_FILES,WINDOWS_LEAF_MAX_PINNED) stay as they are.What changed (motivation → approach → change)
Goal → a selected list should cost each shard about the same. Approach → place the listed files by measured seconds instead of by hash, with the same longest-first rule the pin generator already uses for the full suite. What was built:
scripts/ci_file_shards.py: when--file-shard-targetsis given,balance()places the listed files heaviest first onto the shard with the least time so far. Ties go to the shard holding fewer files, then the lower shard number; equal weights are ordered by path. The placement depends only on the set of targets, the weights and the shard count, so every--file-shardindex and every xdist worker computes the same one. Weights are read as whole milliseconds so no float sum can break a tie differently on two shards. A target with no weight gets the weights file's median. Pins never apply to a target. A collected file that is not a target keeps its pin or hash owner.--file-shard-weights=PATH, default the checked-in file for the platform (sys.platform == "win32"reads the Windows file, everything else the Linux one). The default is what lets this take effect with no workflow change.file-shard targets: <N> file(s), sha256 <digest>; shard <i>/<n> owns <k>in its terminal summary, and the new--file-shard-targets-sha256=<digest>refuses a list with another digest before anything is collected. No workflow passes the option yet (that needs a.github/change), so today a mismatch is visible, not refused.scripts/ci_file_shard_pins.pyalso writes the per-file seconds it already computes to a weights file,<seconds> <path>per line. A default run writes it next to the pin file; a run with a custom--outwrites none unless--weights-outnames one, so a trial plan cannot overwrite the checked-in weights.scripts/ci_file_shard_weights.txt(8,299 files) andscripts/ci_file_shard_weights_windows.txt(8,228 files), generated by that script, not by hand, from thelinux-andwindows-pytest-progress-<N>artifacts of three greenmainruns of 2026-11-21: 44875065432, 44871779180, 44869390554. The pin files were written to a scratch path in the same runs and are byte-identical to main.testing-conventions.md§ Shard balance andci-and-reviews.md§ Backend file sharding said the reduced and leaf scopes keep pure hash ownership.Decisions the issue left open, and what I chose:
weights()(median share of a run's total, scaled by the median total), written beside the pin files. No second measuring path..github/change and the workflows pass no weights flag.What this takes effect on, with no workflow change: the Linux and Windows reduced scope, the leaf corpus gates on both, and the docs contract readers.
Slowest shard, before and after, on 10 recent Windows lists
The acceptance asks for 10 recent reduced-scope Windows target lists. The lists themselves cannot be read back from a run: the Windows job logs print only the count. So each list below is reconstructed by running the workflow's own selector at the PR's squash commit on main (
ci-surface-tests.py --surface backend --summaryminustest/windows-collect-ignore.txtfor the reduced scope,leaf_test_scope.py --gatesfor the leaf scope), and checked against the count the PR's real Windows shard-1 log printed. Two kinds of list exist today and both follow from the checkout, not from the diff, so the rows of one kind are almost the same list.These are estimates, not measurements of a reduced run. Each figure is the sum of the listed files' weights on a shard: test seconds (setup, call, teardown, summed over xdist workers) as measured in the three full-suite Windows runs above. "Before" is the path hash, which is what main does for a targets list; "after" is this PR's placement. A file Windows never collects counts as 0, an unmeasured one as the median.
The 1.00x is in-sample: the same weights place the files and score the result. Held out (weights from two runs, scored on the third run's measured per-file seconds, all three folds), the slowest shard goes from 1.38x to 1.02x-1.04x of the mean on the reduced list (1,547-1,571 s → 1,162-1,182 s) and from 1.48x-1.52x to 1.02x-1.04x on the leaf gates (1,904-1,994 s → 1,316-1,362 s). That is still a replay of full-suite timings, not a reduced run.
Limits to read with it:
changesoutput or from the same selector on the same commit. This PR makes a mismatch visible (the summary line above) and refusable (--file-shard-targets-sha256), but nothing refuses it until a workflow passes one digest computed once per run. That is ci: refuse a backend shard whose selected test-file list differs from the run's #40363, outside this PR because it changes.github/.Backwards compatibility
Compatible: nothing main accepts is rejected. A targets list is still collected exactly once across the shards; only which shard owns a listed file changes.
--file-shard-weightsis new, and is refused without--file-shard-targets, as is the new--file-shard-targets-sha256.read_targetsnow returns an ordered tuple instead of a frozenset; its only caller is this module.Removes nothing: no option, file, key or public function is removed or renamed, and the full-suite path reads the same pins and hashes.
Tests
In
test/test_ci_file_shards.py, all on properties of the placement, none on workflow text:test_balance_gives_each_target_one_shard_within_the_longest_first_bound: 300 seeded cases of targets, weights (zero, equal, missing, far heavier) and 1-12 shards. Each target gets exactly one shard in range; the slowest shard is within the longest-first bound (max <= total/m + (1 - 1/m) * heaviest); themheaviest targets land onmdifferent shards; order and repeats in the list do not change the result; an unmeasured target is placed exactly as one weighing the median.test_balance_spreads_targets_evenly_when_no_weight_tells_them_apart: no weights, none measured, all zero, all equal: shard sizes differ by at most one.test_every_shard_index_and_path_flavour_computes_the_same_target_owners: every--file-shardindex, under a POSIX and a Windows root, with pins naming every target, yields the same owners; a backslash-spelled Windows target is the same file; a target outside the root is refused.test_shard_indices_split_a_target_list_with_no_gap_and_no_file_owned_twice: for generated lists drawn from both checked-in weights files, each with its pin file, the files owned across all 8 shard indices add up to the list, no gap and none owned twice. Covers unmeasured targets, repeats in the input, pinned files as targets, a single target and more shards than targets.test_no_targets_list_leaves_every_file_on_its_pin_or_hash_shard: no targets, so no placement exists and each file's owner is its pin or its hash.test_owner_follows_the_placement_for_targets_and_pin_or_hash_otherwise: in-process. Two heavy files that hash to one shard are owned by different shards; every shard index gives each target the placement's owner; pins that name the targets change nothing; a file that is not a target keeps its pin or hash; every shard's summary line carries the same count and digest.test_targets_digest_names_the_set_and_nothing_else: the digest ignores order and repeats, changes when a file is added or dropped, is the same under a POSIX and a Windows root, and equals the documentedsort -u | sha256sumspelling.test_real_collection_obeys_the_placement_and_refuses_what_it_cannot_trust: real pytest collection, all scenarios in one interpreter child, in its ownxdist_group. Each shard collects exactly its share and the union is every target once, with explicit weights, with pins added, with the checked-in weights, and with a matching digest. Refused with exit 4: seven malformed weights files, a list whose digest is not the expected one, and weights or a digest without a targets list. A targets file of blank lines exits 5, never the whole suite.test_committed_weights_files_parse_for_both_platforms.test_pin_generator_writes_the_weights_it_measured_and_only_where_asked: the weights file holds what was measured, skips a file with no weight, is byte-stable across runs, and a trial--outleaves the checked-in weights alone.test_reduced_backend_scope_populates_every_ci_shardnow places the reduced list the way the plugin does.Checked by reverting: with
owner()ignoring the placement the first and third of these fail; with the digest comparison removed the third fails.Run here at 9d58200:
test/test_ci_file_shards.py, 40 passed under--dist loadgroup.scripts/local-gate.py: 1,827 passed, 8 skipped, exit 0 (the frontend set is not runnable on this host; CI runs it). black, isort, flake8, the comment-history, brand, line-cap and docs lints, andtest_flake_pattern_ratchet.pypass.Manual verification
No targets, nothing changes: main's module (438cee9) and this PR's were loaded side by side and asked for the owner of all 13,441 tracked
.pyfiles undertest/andsrc/, for all 8 shard indices, with the Linux pin file, the Windows pin file and no pins. Owner differences: 0 in each of the three. Both pin files are byte-identical to main.Real-machine verification
Machine: own computer (a development host), Amazon Linux 2023, x86_64, 32 vCPU / 123 GB.
Before (main 438cee9):
python -m pytest -p scripts.ci_file_shards --collect-only -qqq --no-cov -n0 --file-shards 8 --file-shard <1..8> --file-shard-targets=<the 1,361-file Windows reduced list>-> 1,357 files collected once each across the 8 shards (4 collect nothing on Linux), owners equal to the path hash; summed Windows weights per shard 1,550 / 1,425 / 619 / 647 / 1,343 / 1,332 / 1,180 / 936 s.After (head 6203da3, the first commit; the second changes no placement): the same command plus
--file-shard-weights=scripts/ci_file_shard_weights_windows.txt(this host is Linux, so the Windows file is named) -> the same 1,357 files once each, owners equal tobalance(); per shard 1,130 / 1,129 / 1,130 / 1,130 / 1,129 / 1,130 / 1,130 / 1,128 s. Items: 66,645 before, 66,654 after; the 9 more are this PR's new tests.Cleanup: own computer; nothing was started beyond pytest collection.
Not run: a reduced or leaf run on a Windows runner with this placement. This PR's own CI is a full run (it touches
scripts/), so the first measured figure will come from the next frontend-only or leaf PR after merge.Related Issues
Closes #40160. Part of #39690. Needed before #39689 turns on a selected Windows set.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commitsContribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.
AI-assisted, opened by Claude Code on behalf of Zedmor