Skip to content

ci: spread a selected test-file set across shards by measured time - #40348

Merged
Zedmor merged 3 commits into
mainfrom
ci/balance-selected-shards-40160
Nov 21, 2026
Merged

Zedmor merged 3 commits into
mainfrom
ci/balance-selected-shards-40160

Conversation

@Zedmor

@Zedmor Zedmor commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A reduced-scope backend run finishes when its work is done, because the selected test files are spread across the shards by measured time and no shard is left holding the heavy ones.

scripts/ci_file_shards.py owns a --file-shard-targets list by path hash. That evens out the full suite, where the pin files correct it, but not a small selected set: two heavy files that hash to one shard set the run's time while the other shards finish early. On the Windows lists CI runs today the slowest shard carries 1.37x to 1.50x the mean test time (table below).

Why it matters

Diff-based test selection (#39690) is modelled to cut the slowest Windows shard from 22.7 to 7.7 minutes. That model assumes the selected files are spread by time. Without this the saving stops at whatever the unluckiest hash shard costs, and #39689 cannot turn on a selected Windows set and expect the modelled wall time.

Not a goal

  • Changing how the full suite is assigned. With no --file-shard-targets the pins and hashes are as they are on main, and the two pin files are untouched.
  • Any change under .github/. The workflows already pass --file-shard-targets; the Windows leaf caps (WINDOWS_LEAF_MAX_FILES, WINDOWS_LEAF_MAX_PINNED) stay as they are.
  • Spreading the leaf arm's changed files. They run unsharded on every shard by design and never reach this plugin.
  • Refreshing the weights on a schedule. They are regenerated by hand with the pin files, as today.

What changed (motivation → approach → change)

Goal → a selected list should cost each shard about the same. Approach → place the listed files by measured seconds instead of by hash, with the same longest-first rule the pin generator already uses for the full suite. What was built:

  • scripts/ci_file_shards.py: when --file-shard-targets is given, balance() places the listed files heaviest first onto the shard with the least time so far. Ties go to the shard holding fewer files, then the lower shard number; equal weights are ordered by path. The placement depends only on the set of targets, the weights and the shard count, so every --file-shard index and every xdist worker computes the same one. Weights are read as whole milliseconds so no float sum can break a tie differently on two shards. A target with no weight gets the weights file's median. Pins never apply to a target. A collected file that is not a target keeps its pin or hash owner.
  • New option --file-shard-weights=PATH, default the checked-in file for the platform (sys.platform == "win32" reads the Windows file, everything else the Linux one). The default is what lets this take effect with no workflow change.
  • A guard on the list. A placement holds only if every shard of a run is given the same set of targets. No shard can see another's list, so each prints file-shard targets: <N> file(s), sha256 <digest>; shard <i>/<n> owns <k> in its terminal summary, and the new --file-shard-targets-sha256=<digest> refuses a list with another digest before anything is collected. No workflow passes the option yet (that needs a .github/ change), so today a mismatch is visible, not refused.
  • scripts/ci_file_shard_pins.py also writes the per-file seconds it already computes to a weights file, <seconds> <path> per line. A default run writes it next to the pin file; a run with a custom --out writes none unless --weights-out names one, so a trial plan cannot overwrite the checked-in weights.
  • scripts/ci_file_shard_weights.txt (8,299 files) and scripts/ci_file_shard_weights_windows.txt (8,228 files), generated by that script, not by hand, from the linux- and windows-pytest-progress-<N> artifacts of three green main runs of 2026-11-21: 44875065432, 44871779180, 44869390554. The pin files were written to a scratch path in the same runs and are byte-identical to main.
  • Docs: testing-conventions.md § Shard balance and ci-and-reviews.md § Backend file sharding said the reduced and leaf scopes keep pure hash ownership.

Decisions the issue left open, and what I chose:

  1. Where weights come from: the pin generator's own weights() (median share of a run's total, scaled by the median total), written beside the pin files. No second measuring path.
  2. Which weights file a run reads: the platform's, chosen by the plugin, because the acceptance forbids a .github/ change and the workflows pass no weights flag.
  3. Median for an unmeasured target: the median of the whole weights file (1.027 s on Windows), not of the targets, so it does not move with the list.
  4. Pins and targets together: accepted as before; the pins are ignored for a target. No workflow passes both.

What this takes effect on, with no workflow change: the Linux and Windows reduced scope, the leaf corpus gates on both, and the docs contract readers.

Slowest shard, before and after, on 10 recent Windows lists

The acceptance asks for 10 recent reduced-scope Windows target lists. The lists themselves cannot be read back from a run: the Windows job logs print only the count. So each list below is reconstructed by running the workflow's own selector at the PR's squash commit on main (ci-surface-tests.py --surface backend --summary minus test/windows-collect-ignore.txt for the reduced scope, leaf_test_scope.py --gates for the leaf scope), and checked against the count the PR's real Windows shard-1 log printed. Two kinds of list exist today and both follow from the checkout, not from the diff, so the rows of one kind are almost the same list.

These are estimates, not measurements of a reduced run. Each figure is the sum of the listed files' weights on a shard: test seconds (setup, call, teardown, summed over xdist workers) as measured in the three full-suite Windows runs above. "Before" is the path hash, which is what main does for a targets list; "after" is this PR's placement. A file Windows never collects counts as 0, an unmeasured one as the median.

PR Scope Files (reconstructed / logged) Total test s Slowest shard before Slowest shard after Lightest before → after
#40307 reduced 1,361 / 1,361 9,037 1,551 (1.37x mean) 1,130 (1.00x) 619 → 1,130
#40287 reduced 1,361 / 1,361 9,037 1,551 (1.37x) 1,130 (1.00x) 619 → 1,130
#40252 reduced 1,361 / 1,361 9,037 1,551 (1.37x) 1,130 (1.00x) 619 → 1,130
#40171 reduced 1,361 / 1,361 9,037 1,551 (1.37x) 1,130 (1.00x) 619 → 1,130
#40198 reduced 1,361 / 1,361 9,037 1,551 (1.37x) 1,130 (1.00x) 619 → 1,130
#40143 leaf gates 920 / 919 10,316 1,934 (1.50x) 1,290 (1.00x) 811 → 1,289
#40124 leaf gates 918 / 918 10,310 1,930 (1.50x) 1,290 (1.00x) 811 → 1,288
#40031 leaf gates 919 / 919 10,312 1,930 (1.50x) 1,290 (1.00x) 811 → 1,289
#39981 leaf gates 919 / 919 10,312 1,930 (1.50x) 1,290 (1.00x) 811 → 1,289
#39804 leaf gates 917 / 915 10,304 1,930 (1.50x) 1,289 (1.00x) 811 → 1,287

The 1.00x is in-sample: the same weights place the files and score the result. Held out (weights from two runs, scored on the third run's measured per-file seconds, all three folds), the slowest shard goes from 1.38x to 1.02x-1.04x of the mean on the reduced list (1,547-1,571 s → 1,162-1,182 s) and from 1.48x-1.52x to 1.02x-1.04x on the leaf gates (1,904-1,994 s → 1,316-1,362 s). That is still a replay of full-suite timings, not a reduced run.

Limits to read with it:

  • No list here is small. The case the issue describes (a handful of selected files, two of them heavy) does not occur in CI until ci: move the Windows backend shards to diff-based test selection after the Linux gate #39689 selects one; the unit tests cover it, the table cannot.
  • The placement needs every shard of a run to be given the same set of targets. A hash owner held whatever the other shards were given; a placement does not. Today each list comes from one changes output or from the same selector on the same commit. This PR makes a mismatch visible (the summary line above) and refusable (--file-shard-targets-sha256), but nothing refuses it until a workflow passes one digest computed once per run. That is ci: refuse a backend shard whose selected test-file list differs from the run's #40363, outside this PR because it changes .github/.
  • A stale weight costs balance, never a test: weights only move a listed file between shards.

Backwards compatibility

Compatible: nothing main accepts is rejected. A targets list is still collected exactly once across the shards; only which shard owns a listed file changes. --file-shard-weights is new, and is refused without --file-shard-targets, as is the new --file-shard-targets-sha256. read_targets now returns an ordered tuple instead of a frozenset; its only caller is this module.

Removes nothing: no option, file, key or public function is removed or renamed, and the full-suite path reads the same pins and hashes.

Tests

In test/test_ci_file_shards.py, all on properties of the placement, none on workflow text:

  • test_balance_gives_each_target_one_shard_within_the_longest_first_bound: 300 seeded cases of targets, weights (zero, equal, missing, far heavier) and 1-12 shards. Each target gets exactly one shard in range; the slowest shard is within the longest-first bound (max <= total/m + (1 - 1/m) * heaviest); the m heaviest targets land on m different shards; order and repeats in the list do not change the result; an unmeasured target is placed exactly as one weighing the median.
  • test_balance_spreads_targets_evenly_when_no_weight_tells_them_apart: no weights, none measured, all zero, all equal: shard sizes differ by at most one.
  • test_every_shard_index_and_path_flavour_computes_the_same_target_owners: every --file-shard index, under a POSIX and a Windows root, with pins naming every target, yields the same owners; a backslash-spelled Windows target is the same file; a target outside the root is refused.
  • test_shard_indices_split_a_target_list_with_no_gap_and_no_file_owned_twice: for generated lists drawn from both checked-in weights files, each with its pin file, the files owned across all 8 shard indices add up to the list, no gap and none owned twice. Covers unmeasured targets, repeats in the input, pinned files as targets, a single target and more shards than targets.
  • test_no_targets_list_leaves_every_file_on_its_pin_or_hash_shard: no targets, so no placement exists and each file's owner is its pin or its hash.
  • test_owner_follows_the_placement_for_targets_and_pin_or_hash_otherwise: in-process. Two heavy files that hash to one shard are owned by different shards; every shard index gives each target the placement's owner; pins that name the targets change nothing; a file that is not a target keeps its pin or hash; every shard's summary line carries the same count and digest.
  • test_targets_digest_names_the_set_and_nothing_else: the digest ignores order and repeats, changes when a file is added or dropped, is the same under a POSIX and a Windows root, and equals the documented sort -u | sha256sum spelling.
  • test_real_collection_obeys_the_placement_and_refuses_what_it_cannot_trust: real pytest collection, all scenarios in one interpreter child, in its own xdist_group. Each shard collects exactly its share and the union is every target once, with explicit weights, with pins added, with the checked-in weights, and with a matching digest. Refused with exit 4: seven malformed weights files, a list whose digest is not the expected one, and weights or a digest without a targets list. A targets file of blank lines exits 5, never the whole suite.
  • test_committed_weights_files_parse_for_both_platforms.
  • test_pin_generator_writes_the_weights_it_measured_and_only_where_asked: the weights file holds what was measured, skips a file with no weight, is byte-stable across runs, and a trial --out leaves the checked-in weights alone.
  • test_reduced_backend_scope_populates_every_ci_shard now places the reduced list the way the plugin does.

Checked by reverting: with owner() ignoring the placement the first and third of these fail; with the digest comparison removed the third fails.

Run here at 9d58200: test/test_ci_file_shards.py, 40 passed under --dist loadgroup. scripts/local-gate.py: 1,827 passed, 8 skipped, exit 0 (the frontend set is not runnable on this host; CI runs it). black, isort, flake8, the comment-history, brand, line-cap and docs lints, and test_flake_pattern_ratchet.py pass.

Manual verification

No targets, nothing changes: main's module (438cee9) and this PR's were loaded side by side and asked for the owner of all 13,441 tracked .py files under test/ and src/, for all 8 shard indices, with the Linux pin file, the Windows pin file and no pins. Owner differences: 0 in each of the three. Both pin files are byte-identical to main.

Real-machine verification

Machine: own computer (a development host), Amazon Linux 2023, x86_64, 32 vCPU / 123 GB.
Before (main 438cee9): python -m pytest -p scripts.ci_file_shards --collect-only -qqq --no-cov -n0 --file-shards 8 --file-shard <1..8> --file-shard-targets=<the 1,361-file Windows reduced list> -> 1,357 files collected once each across the 8 shards (4 collect nothing on Linux), owners equal to the path hash; summed Windows weights per shard 1,550 / 1,425 / 619 / 647 / 1,343 / 1,332 / 1,180 / 936 s.
After (head 6203da3, the first commit; the second changes no placement): the same command plus --file-shard-weights=scripts/ci_file_shard_weights_windows.txt (this host is Linux, so the Windows file is named) -> the same 1,357 files once each, owners equal to balance(); per shard 1,130 / 1,129 / 1,130 / 1,130 / 1,129 / 1,130 / 1,130 / 1,128 s. Items: 66,645 before, 66,654 after; the 9 more are this PR's new tests.
Cleanup: own computer; nothing was started beyond pytest collection.

Not run: a reduced or leaf run on a Windows runner with this placement. This PR's own CI is a full run (it touches scripts/), so the first measured figure will come from the next frontend-only or leaf PR after merge.

Related Issues

Closes #40160. Part of #39690. Needed before #39689 turns on a selected Windows set.

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

AI-assisted, opened by Claude Code on behalf of Zedmor

A --file-shard-targets list was owned by path hash, which evens out the
whole suite but not a small part of it: two heavy files sharing a shard
set the run's time while the other shards finish early.

The plugin now places the listed files heaviest first onto the shard
with the least time so far, from a per-platform weights file that
scripts/ci_file_shard_pins.py writes next to the pin files. A listed
file with no weight gets the median. Every shard computes the same
placement from the list, the weights and the shard count.

With no targets list nothing changes: the full suite keeps its pins and
hashes, and the pin files are untouched.

Closes #40160
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 9d58200faaae5056c99939dc4cb92e8ba7c0cccc — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The diff only touches CI scripts, weight data files, tests and docs. It adds or changes no user-visible control, so no screenshot is needed.

[UX-REVIEWED] 9d58200

Description read

[DESCRIPTION-READ] 5ae44f35ab88a9834b0f048be25ab4b59d6e87c9a27ed150eb518f1f95f32bd0

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 9d58200faaae5056c99939dc4cb92e8ba7c0cccc — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the placement only moves a listed file between shards, a stale weights file costs balance but never drops a test, and a revert puts back the hash behaviour exactly.

Notes

  • Each shard rebuilds the reduced list itself (ci.yml:1676), and the placement now depends on the whole list. If two shards ever get different lists, files can drop silently, where the old hash design dropped at most the files that differed. The digest is printed but nothing refuses a mismatch yet. Follow-up: build the list once per run and pass --file-shard-targets-sha256.
  • I didn't check the backwards-compatibility section because the description is cut off at 8000 bytes. The one hash reader outside the patch, scripts/ci_rewrite_prewarm.py, is skipped on the reduced scope (ci.yml:1702), so it isn't affected.

[DESIGN-REVIEWED] 9d58200

Description read

[DESCRIPTION-READ] 1d821500acdad100682d769b76598ebde8dd1db43aad442cff48d489971123f2

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 9d58200faaae5056c99939dc4cb92e8ba7c0cccc — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: the spread-by-measured-time fix targets the real cause (hash ownership on a selected list) and keeps the full suite as it was. Two flags have no workflow consumer.

Not checked: whether the main-red-watch and website hunks in the patch are this PR's or drift from its base.

Notes

  • Item 4 — zero consumers: --file-shard-targets-sha256 has 0 consumers under .github/ (searched file-shard-targets-sha256|targets_digest: only the script, the spec and the tests). The description says "No workflow passes the option yet." Subtraction: delete the option and its UsageError branch, and keep only the summary line until a workflow can pass one digest per run.
  • Item 5 — zero consumers: --file-shard-weights=PATH has 0 callers (same search). Every run uses default_weights_path(). Subtraction: delete the option and read default_weights_path() directly.
  • Item 1 — duplicate of scripts/ci_file_shard_pins.py:place: balance() repeats its heaviest-first heap loop. Subtraction: have place() call balance() for its pinned slice, with the hash preload as the starting heap, so there is one greedy loop.

What this change ships

Inventory (7 items) — 4 justified

Intent: make a reduced-scope backend CI run end when its work is done, by spreading the selected test files across shards by measured time. FIX.

  1. Selected target files are placed on shards heaviest-first by measured seconds, not by hash or pins — justified
  2. Two checked-in per-platform weights files (8,299 and 8,228 files) — justified
  3. The pin generator also writes the weights file (--weights-out for trial runs) — justified
  4. New --file-shard-targets-sha256 refusal flag — zero consumers; no workflow passes it
  5. New --file-shard-weights override flag — zero consumers; the default is always used
  6. Each shard prints a targets digest/count line in its terminal summary — justified
  7. Spec text in testing-conventions and ci-and-reviews updated — justified

[FIRST-PRINCIPLES-REVIEWED] 9d58200

Description read

[DESCRIPTION-READ] 919fea3395bfb8528e35e30e962ab92a2de683103c259d83378218ff87f7b38f

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 9d58200faaae5056c99939dc4cb92e8ba7c0cccc — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9d58200

Verdict parsed from the review's SHA-scoped output markers for commit 9d58200faaae5056c99939dc4cb92e8ba7c0cccc.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9d58200faaae5056c99939dc4cb92e8ba7c0cccc false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 9d58200faaae5056c99939dc4cb92e8ba7c0cccc and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9d58200

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9d58200faaae5056c99939dc4cb92e8ba7c0cccc false-positive: <reason, 20 to 500 characters>

The three new target tests each started their own interpreter children,
one of them with nested xdist workers, beside this module's other
subprocess tests. They now check ownership in-process, and one test in
its own xdist group starts a single child that collects every scenario.

A placement holds only if every shard is given the same targets. Each
shard now prints the count and SHA-256 of its target set in the
terminal summary, and --file-shard-targets-sha256 refuses a list with
another digest.
@Zedmor

Zedmor commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 04fbf52 in answer to the GPT 6.1 finding on 6203da3, plus one guard I owed.

Finding (test/test_ci_file_shards.py:527, D8): upheld. The class is "a new test in this module that starts its own interpreter child". Three of this PR's tests were in it (the ones at lines 508, 533 and 553), about 23 children between them, three of those with nested -n2 workers. All three are replaced:

  • ownership, pins-ignored and the fallback for a non-target are now checked in-process through FileShards.owner();
  • real collection keeps one test, test_real_collection_obeys_the_placement_and_refuses_what_it_cannot_trust, which starts a single child that runs every scenario in that one interpreter, in its own xdist_group. No nested xdist workers remain in what this PR adds.

The subprocess tests that were in this file before the PR are unchanged; moving them is outside this PR.

Guard for differing target lists. A placement holds only if every shard is given the same set of targets. A shard cannot see another shard's list, so nothing inside the plugin can refuse a mismatch without a value shared across the run, and sharing one needs a workflow change this PR may not make. What is in this push:

  • each shard prints file-shard targets: <N> file(s), sha256 <digest>; shard <i>/<n> owns <k> in its terminal summary, so differing shards show;
  • --file-shard-targets-sha256=<digest> refuses a list with another digest before collecting (exit 4).

No workflow passes the option yet, so today a mismatch is visible, not refused. Closing it means computing the digest once per run and passing it to every shard; that is a follow-up under .github/.

The PR body's sections from "What changed" down are updated to match; Goal, Why it matters and Not a goal are untouched.

AI-assisted, posted by Claude Code on behalf of Zedmor

For generated target lists drawn from both checked-in weights files,
the files owned across all eight shard indices add up to the list: no
gap and no file owned twice. Covers unmeasured targets, repeats in the
input, pinned files as targets, a single target and more shards than
targets.
@Zedmor

Zedmor commented Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

Pushed 9d58200 as a third plain commit. AGENTS.md caps a PR at two commits and forbids force-push; the Crew Captain asked for this test before arming, so it is added on top instead of rewriting history. The PR is squash-merged.

It adds one test, test_shard_indices_split_a_target_list_with_no_gap_and_no_file_owned_twice: for generated target lists drawn from both checked-in weights files (Linux and Windows, each with its pin file), the files owned across all 8 shard indices add up to the list, with no gap and no file owned twice. The lists include unmeasured targets, repeats in the input, pinned files as targets, a single target and more shards than targets. It passes as written and found no defect; with the placement made to differ per shard index, 72 of its cases fail.

The fail-closed half of the list guard (the workflow passing --file-shard-targets-sha256) is now tracked in #40363.

AI-assisted, posted by Claude Code on behalf of Zedmor

@Zedmor
Zedmor enabled auto-merge (squash) November 21, 2026 08:09
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 21, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 17187 lines, more than the 400 it may approve on its own.

Selected test-file lists (--file-shard-targets) are now spread across shards heaviest-first by measured time, using platform weights files. Every shard and worker computes the same placement: it depends only on the set of files, the weights and the shard count, and ties break deterministically. With no targets list, ownership is unchanged from main (pin or hash). Every workflow list is built from the same commit or one changes-job output, so shards get identical sets. The pin generator writes weights only on a default run or when --weights-out names a file. Tests cover the placement properties, real collection and refusals. CI is green.

Tier: 2, held for a person: reviewer-finding.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 9d58200faa · 17187 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review checked head 9d58200. Every caller of file_shard and hash_shard passes no targets list, so the full suite keeps main's pins and hash ownership. The three jobs that pass --file-shard-targets build the same list on every shard. balance() gives the same answer on every shard and xdist worker, Linux and Windows. The 15 new or changed tests pass, and 11 of 13 single-line mutations of the plugin turn them red.
Well done: weights are whole milliseconds and every tie-break is spelled out, so the placement depends only on the set, and the property tests check the longest-first bound and that no file is skipped or owned twice across all 8 shard indices.
Optional:

  • test/test_ci_file_shards.py:1204 still says the leaf and reduced runs keep pure hash ownership. They are now placed by measured time.
  • DEFAULT_WEIGHTS has no macos key, but --platform choices come from DEFAULT_OUT. If #37422 lands too, a default macOS refresh hits a KeyError after writing the pin file. A test that asserts set(DEFAULT_WEIGHTS) >= set(DEFAULT_OUT) would catch it.
  • For #40363: if some shards fall back to the full suite and the others stay reduced, a target can go unrun, and a digest check alone will not catch it. Making the reduced or full choice once per run would close that.

@Zedmor
Zedmor merged commit 4ca4040 into main Nov 21, 2026
104 of 108 checks passed
@Zedmor
Zedmor deleted the ci/balance-selected-shards-40160 branch November 21, 2026 09:26
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 21, 2026
Zedmor added a commit that referenced this pull request Nov 21, 2026
#40392)

* ci: refuse a backend shard whose test-file list differs from the run's

A --file-shard-targets list is placed across the shards by measured
time, so every shard of a run must hold the same list: shards holding
different lists can each leave a file to another, and that file runs
nowhere on a green run. The plugin could refuse such a shard since
#40348, but no workflow passed it a digest.

The changes job, which runs once per run and is not repeated when one
failed shard is re-run, now names the digest of each list: the leaf
corpus gates, the reduced scope on Linux and on Windows, and the docs
contract readers. Every pytest call that passes a list passes that
digest with it. A shard whose list differs fails before collecting, as
does one that holds a list the run named no digest for, and one that
holds no list while the run has one. A full-suite run passes neither.

ci-surface-tests.py --for-windows prints the Windows list on any host,
so the Linux changes job names the Windows digest from the selector the
Windows shards run. The plugin refuses a value that is not a digest.

Closes #40363

* ci: make the selector's Windows list the same on every host

The changes job names the Windows shards' digest on Linux with
ci-surface-tests.py --for-windows. Two things in the selector followed
the host and could make a Windows shard's list differ from it: the
glob, which ignores case on Windows and not on Linux, and the encoding
of the printed list, which is the code page on Windows.

The Windows list now matches test file names whatever their case by
name, on both hosts, and the selector prints UTF-8. No tracked file is
affected today: none differs from test_*.py by case only, and none has
a non-ASCII name.

A property test reads one synthetic tree as the Linux job does and as a
Windows shard does (CRLF working tree, CRLF list file, a Windows-form
root) and holds the two digests equal.

* ci: match the selector's test file names by name on every list

The POSIX list still left name matching to rglob, whose case rule
follows the host: on Windows it ignores case. The Windows shards run
test_ci_surface_tests.py, and the test that holds the Linux and Windows
digests equal asks for the POSIX list there, so it would have failed on
every Windows shard.

Both lists now match names in the selector: case must match for the
POSIX list, and is ignored for the Windows one. The lists of the real
tree are unchanged on Linux, and a Windows run always takes the Windows
list.

The property test runs each seed under both glob rules.

---------

Co-authored-by: Akim Akimov <zedmor@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: balance a selected test-file set across backend shards by measured time, not fixed pins

2 participants