Repository navigation
ci: split POSIX backend tests into 8 shards to clear the 50-min cap - #11528
Conversation
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify on this PR's own CI run that the worst 8-shard time lands under the cap — the ~40-min estimate rests on the asserted ~33-min fixed per-shard cost. What this change shipsInventory (2 items) — 2 justifiedIntent: stop green backend suites being killed at the 50-min wall (FIX — a shard cancellation with zero failures, #11526 / PR #11460 timings).
Cause-level alternative checked: durations-balanced splitting already has a mechanism ( [FIRST-PRINCIPLES-REVIEWED] 32908cc |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Estimated headroom (~10 min) is under the ~12-min run-to-run swing the deleted comment documented, so cap-kills can recur — and the shard lever is now spent. WatchThe comment this diff deletes recorded "a 28-minute shard reached 99% at 40 on the next run of the same tree" — a ~12-min swing — while the new worst-shard estimate is ~40 min against the 50-min cap. One bad swing (or modest suite growth on a ~33-min non-dividing fixed cost) reproduces #11526, and the same-kind remedy is gone: 8 POSIX + 8 Windows shards already sit at 16 of the ~20 concurrent-job ceiling, so the next escalation is the #8506 durations file or shrinking the fixed cost, not more shards. Suggestions
[DESIGN-REVIEWED] 32908cc |
The POSIX Backend Tests job splits the full suite into 4 count-balanced shards (no .test_durations file is committed, so pytest-split falls back to an even-by-count split) under a 50-minute job cap. The slowest shard rides the wall: on #11460 shard 1 ran 50.2 min and was cancelled at 99% with every test passing and zero failures, while shards 2/3/4 ran 36.3/47.1/43.2 min -- three of four within 7 min of the cap. A cancelled shard reds Coverage Gate and PR Readiness on every full-suite PR. The ~33-min fixed collection+coverage cost per shard does not divide, so raising the cap alone would only defer the same hit; more shards is the lever. This mirrors the Windows backend job, which already runs 8 shards for the identical count-split reason. A committed .test_durations file was declined in #8506 over its rewrite-every-refresh cost, so this follows the shard-count path instead. Refs #11526
abdc0ea to
32908cc
Compare
NicholasRBowers
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: chore / build tooling (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: CI plumbing only -- POSIX backend test matrix split from 4 to 8 shards (group list plus SHARD_COUNT) in .github/workflows/ci.yml, no runtime code touched.
…backend shards pytest-asyncio 0.20.3 walks every registered fixture on every collected test function (O(tests x fixtures)): 87,244 scans x ~1,420 fixtures = 123.8M coroutine checks, 85% of collection under cProfile. Every CI shard paid ~33 min of that before its first test ran, which is why #11528 went to 8 shards. The rootdir conftest now re-runs the scan only after a fixture registration (FixtureManager._register_fixture is the single chokepoint): collection 478 s -> 68 s locally, identical 112,246 node ids. CI: backend-test and backend-test-windows back to 4 shards each, both caps 60 min (measured worst 4-shard shard: Linux 26 min, Windows 39 min).
…backend shards (#11636) pytest-asyncio 0.20.3 walks every registered fixture on every collected test function (O(tests x fixtures)): 87,244 scans x ~1,420 fixtures = 123.8M coroutine checks, 85% of collection under cProfile. Every CI shard paid ~33 min of that before its first test ran, which is why #11528 went to 8 shards. The rootdir conftest now re-runs the scan only after a fixture registration (FixtureManager._register_fixture is the single chokepoint): collection 478 s -> 68 s locally, identical 112,246 node ids. CI: backend-test and backend-test-windows back to 4 shards each, both caps 60 min (measured worst 4-shard shard: Linux 26 min, Windows 39 min). Co-authored-by: Joe Guo <zejiangg@amazon.com>
…irodotdev#11528) The POSIX Backend Tests job splits the full suite into 4 count-balanced shards (no .test_durations file is committed, so pytest-split falls back to an even-by-count split) under a 50-minute job cap. The slowest shard rides the wall: on kirodotdev#11460 shard 1 ran 50.2 min and was cancelled at 99% with every test passing and zero failures, while shards 2/3/4 ran 36.3/47.1/43.2 min -- three of four within 7 min of the cap. A cancelled shard reds Coverage Gate and PR Readiness on every full-suite PR. The ~33-min fixed collection+coverage cost per shard does not divide, so raising the cap alone would only defer the same hit; more shards is the lever. This mirrors the Windows backend job, which already runs 8 shards for the identical count-split reason. A committed .test_durations file was declined in kirodotdev#8506 over its rewrite-every-refresh cost, so this follows the shard-count path instead. Refs kirodotdev#11526
…backend shards (kirodotdev#11636) pytest-asyncio 0.20.3 walks every registered fixture on every collected test function (O(tests x fixtures)): 87,244 scans x ~1,420 fixtures = 123.8M coroutine checks, 85% of collection under cProfile. Every CI shard paid ~33 min of that before its first test ran, which is why kirodotdev#11528 went to 8 shards. The rootdir conftest now re-runs the scan only after a fixture registration (FixtureManager._register_fixture is the single chokepoint): collection 478 s -> 68 s locally, identical 112,246 node ids. CI: backend-test and backend-test-windows back to 4 shards each, both caps 60 min (measured worst 4-shard shard: Linux 26 min, Windows 39 min). Co-authored-by: Joe Guo <zejiangg@amazon.com>
What is the problem?
The POSIX
Backend Testsjob splits the full backend suite into 4 count-balanced shards under a 50-minute job cap. There is no committed.test_durationsfile, so pytest-split falls back to an even-by-count split (per-shard time is not balanced), and the slowest shard rides the wall. On PR #11460, shard 1 ran 50.2 min and was cancelled at[ 99%]with every test passing and zero failures; shards 2/3/4 ran 36.3 / 47.1 / 43.2 min -- three of four within 7 minutes of the cap. A cancelled shard redsCoverage GateandPR Readinesson every full-suite PR.Why this issue matters to the user
This is a board-wide intermittent-cancellation generator, not a per-PR defect. Any full-matrix PR can have its slowest shard killed at the cap with a green suite underneath, producing a red that looks like a test failure but is not one. It costs authors and reviewers rerun cycles chasing a red with no failing test behind it, and it erodes trust in the backend lane's verdict.
How our fix solves it
Symptom:
Backend Tests (3.12, N)cancelled at 50 min ->Coverage Gatereds on the missing shard artifact ->PR Readinessaggregates red.Root cause: the split is even-by-count (no durations file), so the worst shard's wall-clock exceeds the 50-min cap. The dominant cost is a ~33-min fixed collection+coverage-instrumentation phase per shard that does NOT divide when you add shards -- only the ~14-min variable test tail divides. So raising the cap alone would only defer the same hit, while adding shards shrinks the variable tail enough to clear the wall.
Fix: raise POSIX
SHARD_COUNTfrom 4 to 8 (matrixgroup: [1..8]). Estimated worst shard drops from ~50 min to ~40 min, ~10 min under the cap. This follows the Windows backend job, which already runs 8 shards for the identical count-split reason, and avoids committing a.test_durationsfile (declined in #8506 over its rewrite-every-refresh cost). The 50-min cap is deliberately left unchanged so real per-test hangs stay bounded (pytest-timeout bounds individual tests regardless).CI-config only: the sole change is
.github/workflows/ci.yml(SHARD_COUNT, the matrix group list, and the surrounding comments). Coverage Combine already globs shard artifacts (pattern: coverage-shard-*,merge-multiple: true), so 8 artifacts merge with no further change.What tests we did
python -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))"passes (YAML well-formed).SHARD_COUNT: 4in the file is a different (frontend/vitest) job and is untouched; theBackend Tests (3.12, 4)strings insource_providers.py/test_source_providers.pyare name-parsing examples, not count assertions.Any other suggestions on the work
Adding shards raises concurrent-job usage against the ~20-job public-repo ceiling (already noted in the matrix comment). If durations-based balancing is ever revisited (the #8506 tradeoff), a stored
.test_durationswould let the count drop back down and balance by time -- but that is out of scope here.Refs #11526