Skip to content

ci: split POSIX backend tests into 8 shards to clear the 50-min cap - #11528

Merged
NicholasRBowers merged 1 commit into
mainfrom
ci/posix-backend-shards-11526
Sep 17, 2026
Merged

NicholasRBowers merged 1 commit into
mainfrom
ci/posix-backend-shards-11526

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

What is the problem?

The POSIX Backend Tests job splits the full backend suite into 4 count-balanced shards under a 50-minute job cap. There is no committed .test_durations file, so pytest-split falls back to an even-by-count split (per-shard time is not balanced), and the slowest shard rides the wall. On PR #11460, shard 1 ran 50.2 min and was cancelled at [ 99%] with every test passing and zero failures; shards 2/3/4 ran 36.3 / 47.1 / 43.2 min -- three of four within 7 minutes of the cap. A cancelled shard reds Coverage Gate and PR Readiness on every full-suite PR.

Why this issue matters to the user

This is a board-wide intermittent-cancellation generator, not a per-PR defect. Any full-matrix PR can have its slowest shard killed at the cap with a green suite underneath, producing a red that looks like a test failure but is not one. It costs authors and reviewers rerun cycles chasing a red with no failing test behind it, and it erodes trust in the backend lane's verdict.

How our fix solves it

Symptom: Backend Tests (3.12, N) cancelled at 50 min -> Coverage Gate reds on the missing shard artifact -> PR Readiness aggregates red.

Root cause: the split is even-by-count (no durations file), so the worst shard's wall-clock exceeds the 50-min cap. The dominant cost is a ~33-min fixed collection+coverage-instrumentation phase per shard that does NOT divide when you add shards -- only the ~14-min variable test tail divides. So raising the cap alone would only defer the same hit, while adding shards shrinks the variable tail enough to clear the wall.

Fix: raise POSIX SHARD_COUNT from 4 to 8 (matrix group: [1..8]). Estimated worst shard drops from ~50 min to ~40 min, ~10 min under the cap. This follows the Windows backend job, which already runs 8 shards for the identical count-split reason, and avoids committing a .test_durations file (declined in #8506 over its rewrite-every-refresh cost). The 50-min cap is deliberately left unchanged so real per-test hangs stay bounded (pytest-timeout bounds individual tests regardless).

CI-config only: the sole change is .github/workflows/ci.yml (SHARD_COUNT, the matrix group list, and the surrounding comments). Coverage Combine already globs shard artifacts (pattern: coverage-shard-*, merge-multiple: true), so 8 artifacts merge with no further change.

What tests we did

  • python -c "import yaml; yaml.safe_load(open('.github/workflows/ci.yml'))" passes (YAML well-formed).
  • Confirmed no hardcoded 4-shard dependency: Coverage Combine downloads by glob, not a fixed count; the other SHARD_COUNT: 4 in the file is a different (frontend/vitest) job and is untouched; the Backend Tests (3.12, 4) strings in source_providers.py / test_source_providers.py are name-parsing examples, not count assertions.
  • The real validation is CI itself on this PR: the 8-shard split should land the worst POSIX shard comfortably under the 50-min cap.

Any other suggestions on the work

Adding shards raises concurrent-job usage against the ~20-job public-repo ceiling (already noted in the matrix comment). If durations-based balancing is ever revisited (the #8506 tradeoff), a stored .test_durations would let the count drop back down and balance by time -- but that is out of scope here.

Refs #11526

@chenmingwei23
chenmingwei23 requested review from a team and CrysisDeu September 17, 2026 12:05
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 17, 2026
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 32908cc83262976541454372006c45de97ac41a0 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 32908cc

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 32908cc83262976541454372006c45de97ac41a0: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 32908cc83262976541454372006c45de97ac41a0 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 32908cc

Verdict parsed from the review's SHA-scoped output markers for commit 32908cc83262976541454372006c45de97ac41a0.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 32908cc83262976541454372006c45de97ac41a0: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 32908cc83262976541454372006c45de97ac41a0 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify on this PR's own CI run that the worst 8-shard time lands under the cap — the ~40-min estimate rests on the asserted ~33-min fixed per-shard cost.

What this change ships

Inventory (2 items) — 2 justified

Intent: stop green backend suites being killed at the 50-min wall (FIX — a shard cancellation with zero failures, #11526 / PR #11460 timings).

  1. POSIX backend tests now run as 8 parallel shards instead of 4 — justified
  2. Shard and timeout comments rewritten to record the 50.2-min observation and the more-shards-not-higher-cap rationale — justified

Cause-level alternative checked: durations-balanced splitting already has a mechanism (.github/workflows/test-durations.yml) whose own header records the decision NOT to commit the file (~1.8 min gain vs 11.7 MB rewritten per refresh, #8506) — so the count bump is the accepted level, not an unexamined symptom patch. Grepped for hardcoded 4-shard consumers: Coverage Combine globs coverage-shard-* (ci.yml:1568), the other SHARD_COUNT: 4 (ci.yml:2192) is the vitest job; zero POSIX-count dependencies found. No pin reversed: the cap stays 50 and the old comment's headroom principle is restored by the shard count, not deleted.

[FIRST-PRINCIPLES-REVIEWED] 32908cc

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 32908cc83262976541454372006c45de97ac41a0 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Estimated headroom (~10 min) is under the ~12-min run-to-run swing the deleted comment documented, so cap-kills can recur — and the shard lever is now spent.

Watch

The comment this diff deletes recorded "a 28-minute shard reached 99% at 40 on the next run of the same tree" — a ~12-min swing — while the new worst-shard estimate is ~40 min against the 50-min cap. One bad swing (or modest suite growth on a ~33-min non-dividing fixed cost) reproduces #11526, and the same-kind remedy is gone: 8 POSIX + 8 Windows shards already sit at 16 of the ~20 concurrent-job ceiling, so the next escalation is the #8506 durations file or shrinking the fixed cost, not more shards.
Clears when: several full-matrix 8-shard runs show the worst POSIX shard finishing with margin exceeding the documented swing (≤ ~38 min), or a follow-up is filed for the durations-file / fixed-cost path as the next escalation.

Suggestions

  • Record in the matrix comment that 8 is the ceiling for this lever (16/20 concurrent jobs with Windows) and name the durations file as the designated next step, so the next cap-kill doesn't re-litigate "more shards vs higher cap".

[DESIGN-REVIEWED] 32908cc

The POSIX Backend Tests job splits the full suite into 4 count-balanced
shards (no .test_durations file is committed, so pytest-split falls back
to an even-by-count split) under a 50-minute job cap. The slowest shard
rides the wall: on #11460 shard 1 ran 50.2 min and was cancelled at 99%
with every test passing and zero failures, while shards 2/3/4 ran
36.3/47.1/43.2 min -- three of four within 7 min of the cap. A cancelled
shard reds Coverage Gate and PR Readiness on every full-suite PR.

The ~33-min fixed collection+coverage cost per shard does not divide, so
raising the cap alone would only defer the same hit; more shards is the
lever. This mirrors the Windows backend job, which already runs 8 shards
for the identical count-split reason. A committed .test_durations file was
declined in #8506 over its rewrite-every-refresh cost, so this follows the
shard-count path instead.

Refs #11526
@chenmingwei23
chenmingwei23 force-pushed the ci/posix-backend-shards-11526 branch from abdc0ea to 32908cc Compare September 17, 2026 16:08
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026
@NicholasRBowers
NicholasRBowers enabled auto-merge (squash) September 17, 2026 17:24

@NicholasRBowers NicholasRBowers left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: chore / build tooling (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: CI plumbing only -- POSIX backend test matrix split from 4 to 8 shards (group list plus SHARD_COUNT) in .github/workflows/ci.yml, no runtime code touched.

@NicholasRBowers
NicholasRBowers merged commit 94c59ed into main Sep 17, 2026
86 checks passed
@NicholasRBowers
NicholasRBowers deleted the ci/posix-backend-shards-11526 branch September 17, 2026 17:24
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 17, 2026
iamwhatever pushed a commit that referenced this pull request Sep 18, 2026
…backend shards

pytest-asyncio 0.20.3 walks every registered fixture on every collected
test function (O(tests x fixtures)): 87,244 scans x ~1,420 fixtures =
123.8M coroutine checks, 85% of collection under cProfile. Every CI shard
paid ~33 min of that before its first test ran, which is why #11528 went
to 8 shards. The rootdir conftest now re-runs the scan only after a
fixture registration (FixtureManager._register_fixture is the single
chokepoint): collection 478 s -> 68 s locally, identical 112,246 node ids.

CI: backend-test and backend-test-windows back to 4 shards each, both
caps 60 min (measured worst 4-shard shard: Linux 26 min, Windows 39 min).
chenmingwei23 pushed a commit that referenced this pull request Sep 18, 2026
…backend shards (#11636)

pytest-asyncio 0.20.3 walks every registered fixture on every collected
test function (O(tests x fixtures)): 87,244 scans x ~1,420 fixtures =
123.8M coroutine checks, 85% of collection under cProfile. Every CI shard
paid ~33 min of that before its first test ran, which is why #11528 went
to 8 shards. The rootdir conftest now re-runs the scan only after a
fixture registration (FixtureManager._register_fixture is the single
chokepoint): collection 478 s -> 68 s locally, identical 112,246 node ids.

CI: backend-test and backend-test-windows back to 4 shards each, both
caps 60 min (measured worst 4-shard shard: Linux 26 min, Windows 39 min).

Co-authored-by: Joe Guo <zejiangg@amazon.com>
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…irodotdev#11528)

The POSIX Backend Tests job splits the full suite into 4 count-balanced
shards (no .test_durations file is committed, so pytest-split falls back
to an even-by-count split) under a 50-minute job cap. The slowest shard
rides the wall: on kirodotdev#11460 shard 1 ran 50.2 min and was cancelled at 99%
with every test passing and zero failures, while shards 2/3/4 ran
36.3/47.1/43.2 min -- three of four within 7 min of the cap. A cancelled
shard reds Coverage Gate and PR Readiness on every full-suite PR.

The ~33-min fixed collection+coverage cost per shard does not divide, so
raising the cap alone would only defer the same hit; more shards is the
lever. This mirrors the Windows backend job, which already runs 8 shards
for the identical count-split reason. A committed .test_durations file was
declined in kirodotdev#8506 over its rewrite-every-refresh cost, so this follows the
shard-count path instead.

Refs kirodotdev#11526
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…backend shards (kirodotdev#11636)

pytest-asyncio 0.20.3 walks every registered fixture on every collected
test function (O(tests x fixtures)): 87,244 scans x ~1,420 fixtures =
123.8M coroutine checks, 85% of collection under cProfile. Every CI shard
paid ~33 min of that before its first test ran, which is why kirodotdev#11528 went
to 8 shards. The rootdir conftest now re-runs the scan only after a
fixture registration (FixtureManager._register_fixture is the single
chokepoint): collection 478 s -> 68 s locally, identical 112,246 node ids.

CI: backend-test and backend-test-windows back to 4 shards each, both
caps 60 min (measured worst 4-shard shard: Linux 26 min, Windows 39 min).

Co-authored-by: Joe Guo <zejiangg@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants