Repository navigation
perf(test): gate pytest-asyncio's per-test fixture rescan; 4 backend shards - #11636
Conversation
|
Intent: Remove the fixed ~33-minute collection cost every backend CI shard pays (pytest-asyncio 0.20.3 rescanning every fixture per test function), so the backend matrix can run on 4 shards per platform instead of 8 without riding its timeout cap. |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the Windows count-split imbalance (worst shard 39 of 60 min) leaves growth headroom until the durations-file fix lands. [DESIGN-REVIEWED] fcb4289 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS POSIX cap 50→60 rests on Windows symmetry — measured worst POSIX shard (26 min) already had 24 min headroom under 50; one "eight shards" prose sibling left stale. Not justified as shipped
What this change shipsInventory (9 items) — 7 justifiedIntent: make backend test collection ~7× faster so CI needs half the backend shards — a FIX (measured performance defect: 85% of collection is a redundant pytest-asyncio rescan) with declared CI-resource changes.
[FIRST-PRINCIPLES-REVIEWED] fcb4289 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
|
Round 1 on
Neither test touches Shard wall times on this head, 4 shards each (the point of the PR):
Half the runners, each shard finishes sooner than the 8-shard median did, ~35 min of headroom under the 60-min cap. |
…backend shards pytest-asyncio 0.20.3 walks every registered fixture on every collected test function (O(tests x fixtures)): 87,244 scans x ~1,420 fixtures = 123.8M coroutine checks, 85% of collection under cProfile. Every CI shard paid ~33 min of that before its first test ran, which is why #11528 went to 8 shards. The rootdir conftest now re-runs the scan only after a fixture registration (FixtureManager._register_fixture is the single chokepoint): collection 478 s -> 68 s locally, identical 112,246 node ids. CI: backend-test and backend-test-windows back to 4 shards each, both caps 60 min (measured worst 4-shard shard: Linux 26 min, Windows 39 min).
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
ddcf210 to
fcb4289
Compare
|
|
…backend shards (kirodotdev#11636) pytest-asyncio 0.20.3 walks every registered fixture on every collected test function (O(tests x fixtures)): 87,244 scans x ~1,420 fixtures = 123.8M coroutine checks, 85% of collection under cProfile. Every CI shard paid ~33 min of that before its first test ran, which is why kirodotdev#11528 went to 8 shards. The rootdir conftest now re-runs the scan only after a fixture registration (FixtureManager._register_fixture is the single chokepoint): collection 478 s -> 68 s locally, identical 112,246 node ids. CI: backend-test and backend-test-windows back to 4 shards each, both caps 60 min (measured worst 4-shard shard: Linux 26 min, Windows 39 min). Co-authored-by: Joe Guo <zejiangg@amazon.com>
Problem / Motivation
Every backend CI shard spends ~33 minutes before its first test runs. On run 35279898889 (main, 2026-09-17) shard 1 started pytest at 23:25 and printed
test session startsat 23:58, then ran its 14,173 tests in 10 minutes. The Windows shards show the same shape without coverage: workers ready 23:12, collected 23:22.That fixed cost is why #11528 split the POSIX backend job into 8 shards (and #10868 did the same on Windows): a cost that does not divide can only be hidden behind more runners. The price is 16 backend jobs per full-matrix PR. At peak they hold the runner pool and every other PR queues behind them.
Why it matters
A median CI run is ~240 job-minutes and the backend shards are ~74% of it; most of that is collection nobody needs. Fewer, faster shards cut the queue for every PR in flight, and the suite gets the same speed-up on every developer's machine:
pytest --collect-onlyon the full suite goes from 8 minutes to 1.What changed (motivation → approach → change)
The slow part is pytest-asyncio 0.20.3, not the tests. Its
pytest_pycollect_makeitemhook calls_preprocess_async_fixturesfor every test function it sees, and that function walks every fixture definition the session has registered so far to find async ones. Wrapped fixtures are skipped by a set lookup; the ~1,400 synchronous ones are re-inspected withasyncio.iscoroutinefunctioneach time. Under cProfile that was 87,244 scans × ~1,420 fixtures = 123.8 million checks: 1,098 of 1,285 profiled seconds, 85% of collection. Every xdist worker pays it in full, and coverage's tracer makes each check ~2.3× dearer, which is the 33-minute CI number. Upstream fixed it only in 0.25.1 / 1.0.0, behind the v0.23 event-loop rework this suite has not migrated to.The scan's result only changes when a fixture is added, and pytest funnels every registration through
FixtureManager._register_fixture. The rootdirconftest.pynow wraps that one method to raise a dirty flag and replaces the plugin's scan function with a gate (_AsyncFixtureScanGate) that lets the scan run only while the flag is up. Nothing else about the scan changes: same arguments, same_HOLDERset, same wrapping. Both seams are private, so if a future pytest or pytest-asyncio renames one the gate steps aside with aRuntimeWarningand collection is merely slow again, never a crash before collection.With the fixed cost gone,
ci.ymlreturns both backend jobs to 4 shards. Both caps go to 60 min: on this PR's first 4-shard run the worst Linux shard took 26 min and the worst Windows shard 39 min (Windows count-split hands one shard the slow files), so each keeps 20+ min of headroom instead of riding the wall; pytest-timeout still bounds each test.test_ci_pytest_progress.pypins the new Windows values anddocs/ci/ci-and-reviews.mdstops calling the shards "eight".Tests
test/test_pytest_asyncio_scan_gate.py(10 tests, 8 red on main):FixtureManager._register_fixtureraises the flag, installation is idempotent, and collection ran far fewer scans than test functions@pytest_asyncio.fixtureregistered after the gate went clean is still wrapped (pytester, in-process): this suite has no async fixtures of its own, so the wrap is exercised through an inner sessionEquivalence:
--collect-onlyon main vs this branch produced byte-identical sorted node-id lists (112,246 lines,diffempty).test_ci_pytest_progress.pyupdated for the Windows matrix.Manual verification
Timings above were measured locally with
pytest --collect-only -q -n0 -o addopts= -p no:covon this checkout vs a detached worktree at main. The CI effect (shard wall time at 4 shards) is verified by this PR's own run.Related Issues
Refs #11528, #10868, #10523.
Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)