Skip to content

test(ci): make shard-digest leaf sampling ignore pinned files - #40565

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
test/shard-digest-sampling-ignore-pins
Nov 22, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
test/shard-digest-sampling-ignore-pins

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A PR that adds test files no longer turns test_ci_shard_list_digest.py red for a reason unrelated to its own change.

test/test_ci_shard_list_digest.py::test_a_shard_holding_no_list_fails_when_the_run_has_one[windows leaf gates] (added in #40392) does not read a real diff. It takes the first 400 files of test/test_*.py by name and samples 12 of them with a fixed seed as the "changed files". When a PR adds test files that sort near the front (for example three test/test_acp_*.py files), the window shifts and the sample changes. If the new sample holds more than WINDOWS_LEAF_MAX_PINNED=4 files listed in scripts/ci_file_shard_pins_windows.txt, the Windows scope step correctly falls back to the full suite and exits 0, and the test asserting code != 0 fails. On current main, adding three empty files test/test_acp_a1.py..a3.py moves the sample from 2 pinned files to 6.

Why it matters

Any contributor whose PR adds test files can get a red Backend Tests shard from a test that has nothing to do with their change, and the failure message (a Windows budget notice) points them at the CI script instead of at the sampling. This already happened on an open PR.

Not a goal

  • No change to the CI workflow steps, WINDOWS_LEAF_MAX_PINNED, or either pin file.
  • No change to what the digest tests assert.

What changed (motivation -> approach -> change)

Symptom: the windows leaf-gates case fails when early-sorting test files are added. Root cause: the sampled corpus includes Windows-pinned files, so the pinned count in the sample (and with it the scope step outcome) depends on which files sort first in test/. Change: the corpus fixture now goes through a small _corpus() helper that drops every path listed in scripts/ci_file_shard_pins_windows.txt (parsed with the plugin own ci_file_shards.read_pins, shard count read from the Windows job SHARD_COUNT) before taking the first 400. Every sampling test in the file uses that fixture, so all of them are covered by the one change; none of them needs a pinned file.

Backwards compatibility

Compatible: test-only change; no runtime or workflow code is touched.

Removes nothing: only the test corpus fixture is narrowed, and no reader outside this test file uses it.

Tests

  • New test_new_test_files_sorting_first_leave_the_windows_leaf_sample_failing[0..5]: adds 0 to 5 file names that sort before every real test file, rebuilds the corpus through _corpus(), and runs the windows leaf-gates scenario of test_a_shard_holding_no_list_fails_when_the_run_has_one. With the pin filter removed from _corpus(), case [3] fails with leaf diff outside the Windows budget (13 ... changed file(s), 6 pinned; need 1-40 and <= 4); with it, all pass.

Proof: not provable -- test-only change

Determinism: the whole file (38 cases) passed 3 times in a row serially, and once more in a seeded shuffled order of all 38 node ids.

Manual verification

N/A -- the behavior is the test outcome itself, covered above.

Real-machine verification

Machine: own computer, Amazon Linux 2023, x86_64, 64 vCPU / 123 GiB
Before (main 378ec94): add empty test/test_acp_a1.py, test/test_acp_a2.py, test/test_acp_a3.py, then pytest -n 0 test/test_ci_shard_list_digest.py -k no_list_fails -> 1 failed, 4 passed ([windows leaf gates]: 6 pinned; need <= 4)
After (head b03d565): same three files, pytest -n 0 test/test_ci_shard_list_digest.py -> 38 passed
Cleanup: own computer; the three probe files deleted

Related Issues

Found while a PR adding three test/test_acp_*.py files went red on this case; follow-up to #40392.

Sibling sweep

Swept: git grep -n "random.Random" -- test/test_ci_shard_list_digest.py
Sibling: test/test_ci_shard_list_digest.py -- changed here (all sampling sites read the one corpus fixture)

Pattern harvest

Not generalizable: the hazard is specific to a test that samples real repo files and feeds them to a step with a count budget over a second file list.

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project Apache-2.0 license.

The Windows leaf gate runs the full suite when a diff changes more than
WINDOWS_LEAF_MAX_PINNED pinned files. The digest tests sample their
changed files from the first 400 test files by name, so a PR adding test
files that sort early could shift the sample onto pinned files and turn
test_a_shard_holding_no_list_fails_when_the_run_has_one[windows leaf
gates] red. Sample only from files the Windows pin file does not list,
and add a test that adds leading files and expects the same result.
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision b03d565534066b423987d3cc402d7bb5c4f5aaa2 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision b03d565534066b423987d3cc402d7bb5c4f5aaa2 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of b03d565534066b423987d3cc402d7bb5c4f5aaa2 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a test-only change that removes the order-dependence where it starts, in the sampled corpus.

Notes

  • A pin file growing past the corpus can still shrink the sample window. The new [0..5] cases guard only against files that sort first, not against the pin list growing.

[DESIGN-REVIEWED] b03d565

Description read

[DESCRIPTION-READ] 43e278659f924de9703dc3e81349f70df1bb1cbf710a67ef7f15eafe1051e251

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed b03d565534066b423987d3cc402d7bb5c4f5aaa2 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] b03d565

Verdict parsed from the review's SHA-scoped output markers for commit b03d565534066b423987d3cc402d7bb5c4f5aaa2.

False positive or not applicable? A repository writer can comment:
/ai-review override fable b03d565534066b423987d3cc402d7bb5c4f5aaa2 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of b03d565534066b423987d3cc402d7bb5c4f5aaa2 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] b03d565

False positive or not applicable? A repository writer can comment:
/ai-review override gpt b03d565534066b423987d3cc402d7bb5c4f5aaa2 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 22, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 22, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 5% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

Test-only change: the corpus fixture now excludes Windows-pinned files (parsed with ci_file_shards.read_pins, shard count 8 from the backend-test-windows job env), so a sample can no longer exceed WINDOWS_LEAF_MAX_PINNED and push the windows leaf-gates case onto the full-suite fallback. The workflow's targets-list paths never apply pins (ci.yml:2296), so excluding pinned files loses no coverage. The fixture scopes match, the unpinned pool is still well over 400 files, and the new parametrised regression test reaches the same empty-gates ::error:: path. CI is green.

Tier: 1, every tier gate passed.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit b03d565534 · 28 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 22, 2026 03:53

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved automatically by the code change approver, not by a person. The decider estimates a 5% chance that this change introduces a defect, within the 12% limit for approving on its own, and the change has at most 400 changed lines.

Test-only change: the corpus fixture now excludes Windows-pinned files (parsed with ci_file_shards.read_pins, shard count 8 from the backend-test-windows job env), so a sample can no longer exceed WINDOWS_LEAF_MAX_PINNED and push the windows leaf-gates case onto the full-suite fallback. The workflow's targets-list paths never apply pins (ci.yml:2296), so excluding pinned files loses no coverage. The fixture scopes match, the unpinned pool is still well over 400 files, and the new parametrised regression test reaches the same empty-gates ::error:: path. CI is green.

Tier: 1, every tier gate passed.
Criteria: no merge conflict, no owner block, no change request, the default branch, no security-sensitive path, no design document to agree on, the readiness status, the review lanes, the P0 history, the linked issues' tier, the review lessons, no concern label, the file count, a safe change category, not its own change, the decider, the security checklist, the change matches its title, no competing change.

Decider v3c (217fe607). To overrule it, request changes or dismiss this review.

@KiroCrewCaptain
KiroCrewCaptain merged commit f4bc834 into main Nov 22, 2026
158 of 160 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the test/shard-digest-sampling-ignore-pins branch November 22, 2026 03:53
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants