Skip to content

perf(test): gate pytest-asyncio's per-test fixture rescan; 4 backend shards - #11636

Merged
chenmingwei23 merged 1 commit into
mainfrom
perf/gate-pytest-asyncio-fixture-scan
Sep 18, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
perf/gate-pytest-asyncio-fixture-scan

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Every backend CI shard spends ~33 minutes before its first test runs. On run 35279898889 (main, 2026-09-17) shard 1 started pytest at 23:25 and printed test session starts at 23:58, then ran its 14,173 tests in 10 minutes. The Windows shards show the same shape without coverage: workers ready 23:12, collected 23:22.

That fixed cost is why #11528 split the POSIX backend job into 8 shards (and #10868 did the same on Windows): a cost that does not divide can only be hidden behind more runners. The price is 16 backend jobs per full-matrix PR. At peak they hold the runner pool and every other PR queues behind them.

Why it matters

A median CI run is ~240 job-minutes and the backend shards are ~74% of it; most of that is collection nobody needs. Fewer, faster shards cut the queue for every PR in flight, and the suite gets the same speed-up on every developer's machine: pytest --collect-only on the full suite goes from 8 minutes to 1.

What changed (motivation → approach → change)

Before After
Full-suite collection, one process, no coverage (local) 478 s 68 s
Node ids collected 112,246 112,246 (identical list)
POSIX backend shards / cap 8 / 50 min 4 / 60 min
Windows backend shards / cap 8 / 40 min 4 / 60 min

The slow part is pytest-asyncio 0.20.3, not the tests. Its pytest_pycollect_makeitem hook calls _preprocess_async_fixtures for every test function it sees, and that function walks every fixture definition the session has registered so far to find async ones. Wrapped fixtures are skipped by a set lookup; the ~1,400 synchronous ones are re-inspected with asyncio.iscoroutinefunction each time. Under cProfile that was 87,244 scans × ~1,420 fixtures = 123.8 million checks: 1,098 of 1,285 profiled seconds, 85% of collection. Every xdist worker pays it in full, and coverage's tracer makes each check ~2.3× dearer, which is the 33-minute CI number. Upstream fixed it only in 0.25.1 / 1.0.0, behind the v0.23 event-loop rework this suite has not migrated to.

The scan's result only changes when a fixture is added, and pytest funnels every registration through FixtureManager._register_fixture. The rootdir conftest.py now wraps that one method to raise a dirty flag and replaces the plugin's scan function with a gate (_AsyncFixtureScanGate) that lets the scan run only while the flag is up. Nothing else about the scan changes: same arguments, same _HOLDER set, same wrapping. Both seams are private, so if a future pytest or pytest-asyncio renames one the gate steps aside with a RuntimeWarning and collection is merely slow again, never a crash before collection.

With the fixed cost gone, ci.yml returns both backend jobs to 4 shards. Both caps go to 60 min: on this PR's first 4-shard run the worst Linux shard took 26 min and the worst Windows shard 39 min (Windows count-split hands one shard the slow files), so each keeps 20+ min of headroom instead of riding the wall; pytest-timeout still bounds each test. test_ci_pytest_progress.py pins the new Windows values and docs/ci/ci-and-reviews.md stops calling the shards "eight".

Tests

test/test_pytest_asyncio_scan_gate.py (10 tests, 8 red on main):

  • the gate scans once when dirty and not again until a registration re-arms it; a scan that raises leaves it dirty
  • in this session the plugin's scan IS the gate, FixtureManager._register_fixture raises the flag, installation is idempotent, and collection ran far fewer scans than test functions
  • a moved private seam warns instead of aborting
  • an @pytest_asyncio.fixture registered after the gate went clean is still wrapped (pytester, in-process): this suite has no async fixtures of its own, so the wrap is exercised through an inner session

Equivalence: --collect-only on main vs this branch produced byte-identical sorted node-id lists (112,246 lines, diff empty). test_ci_pytest_progress.py updated for the Windows matrix.

Manual verification

Timings above were measured locally with pytest --collect-only -q -n0 -o addopts= -p no:cov on this checkout vs a detached worktree at main. The CI effect (shard wall time at 4 shards) is verified by this PR's own run.

Related Issues

Refs #11528, #10868, #10523.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and hoang-phan98 September 18, 2026 01:41
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Intent: Remove the fixed ~33-minute collection cost every backend CI shard pays (pytest-asyncio 0.20.3 rescanning every fixture per test function), so the backend matrix can run on 4 shards per platform instead of 8 without riding its timeout cap.
Not a goal: Upgrading pytest-asyncio past 0.20.3, changing the split strategy (durations file), balancing shard contents, or speeding up individual slow tests.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of fcb42892168343977e58c74cb893f3ac158f70e4 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the Windows count-split imbalance (worst shard 39 of 60 min) leaves growth headroom until the durations-file fix lands.

[DESIGN-REVIEWED] fcb4289

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of fcb42892168343977e58c74cb893f3ac158f70e4 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

POSIX cap 50→60 rests on Windows symmetry — measured worst POSIX shard (26 min) already had 24 min headroom under 50; one "eight shards" prose sibling left stale.

Not justified as shipped

  • Item 4 — inherited: "Both caps go to 60 … each keeps 20+ min of headroom" is the symmetry form; only Windows's measured 39-min shard needed a higher wall, and the deleted POSIX pin ("The cap stays 50 to keep real per-test hangs bounded") loses only its shard-math clause, not its bound-the-hang purpose.
    Subtraction: keep timeout-minutes: 50 on backend-test; raise only the Windows cap.
    Clears when: a measured 4-shard POSIX shard past ~40 min, or the POSIX cap reverted to 50.
  • Item 7 — symptom-level: the PR deletes "eight" from docs/ci/ci-and-reviews.md but grep for eight|8 shards leaves 1 unfixed sibling — src/kiro_crew/builtin_skills/kirocrew-dev/prepare-pr/references/gate-floor.md:166 still says "CI shards that across eight backend runners", the same stale-restated-count trap AGENTS.md pins for the denied-rule count.
    Subtraction: delete "eight" at that line.
    Clears when: the deletion made.

What this change ships

Inventory (9 items) — 7 justified

Intent: make backend test collection ~7× faster so CI needs half the backend shards — a FIX (measured performance defect: 85% of collection is a redundant pytest-asyncio rescan) with declared CI-resource changes.

  1. Collection runs ~7× faster everywhere via the conftest scan gate — justified
  2. If a private seam moves on a version bump, collection warns and stays slow instead of crashing — justified
  3. POSIX backend job drops from 8 shards to 4 — justified
  4. POSIX job timeout rises from 50 to 60 minutes — inherited: symmetry with the Windows cap, no measured need
  5. Windows backend job drops from 8 shards to 4 — justified
  6. Windows job timeout rises from 40 to 60 minutes — justified
  7. CI doc stops calling the shards "eight" — symptom-level: 1 counted sibling restatement left stale in gate-floor.md
  8. New test file pins the gate's behavior (8 red on main) — justified
  9. Windows matrix pins in test_ci_pytest_progress.py updated to the new values — justified

[FIRST-PRINCIPLES-REVIEWED] fcb4289

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed fcb42892168343977e58c74cb893f3ac158f70e4 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fcb4289

Verdict parsed from the review's SHA-scoped output markers for commit fcb42892168343977e58c74cb893f3ac158f70e4.

False positive or not applicable? A repository writer can comment:
/ai-review override fable fcb42892168343977e58c74cb893f3ac158f70e4: <one-sentence reason>

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Round 1 on ddcf210727: the two red shards are both inherited from main, not from this diff.

Red Test Cause on main Status
Backend Tests (3.12, 1) + (Windows) (1) test/test_acp_launch_goldens.py ×10 / ×6: _stub_common.<locals>.<lambda>() got an unexpected keyword argument 'forward_ssh_auth_sock' #9131 added the kwarg to the launch path; the capture stub was not updated issue #11635, fix #11649 open
Backend Tests (3.12, 1) test/test_backend_cards.py::test_every_capability_set_is_classified_by_the_card (ACP_BACKENDS_SELF_SERVED_ACP) #11428 added the set, card not updated fixed on main by e054b09

Neither test touches conftest.py's fixture scan (both fail on a stub signature / a classification table). Plan: rebase once #11649 lands, one push.

Shard wall times on this head, 4 shards each (the point of the PR):

Job Before (8 shards, main run 35279898889) This head (4 shards)
Backend Tests (3.12, N) 24–46 min each, collection ~33 min 21–26 min each, collection ~10 min
Backend Tests (Windows) (N) 17–29 min each, collection ~10 min 20–28 min each, collection ~6 min

Half the runners, each shard finishes sooner than the 8-shard median did, ~35 min of headroom under the 60-min cap.

…backend shards

pytest-asyncio 0.20.3 walks every registered fixture on every collected
test function (O(tests x fixtures)): 87,244 scans x ~1,420 fixtures =
123.8M coroutine checks, 85% of collection under cProfile. Every CI shard
paid ~33 min of that before its first test ran, which is why #11528 went
to 8 shards. The rootdir conftest now re-runs the scan only after a
fixture registration (FixtureManager._register_fixture is the single
chokepoint): collection 478 s -> 68 s locally, identical 112,246 node ids.

CI: backend-test and backend-test-windows back to 4 shards each, both
caps 60 min (measured worst 4-shard shard: Linux 26 min, Windows 39 min).
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of fcb42892168343977e58c74cb893f3ac158f70e4 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fcb4289

False positive or not applicable? A repository writer can comment:
/ai-review override gpt fcb42892168343977e58c74cb893f3ac158f70e4: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the perf/gate-pytest-asyncio-fixture-scan branch from ddcf210 to fcb4289 Compare September 18, 2026 02:33
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • span=0ec13d457dfc — rebutted (POSIX cap stays 60)

The cap's purpose (bound a hang) is served at 60 exactly as at 50: a hung test is killed by --timeout=120 per test, and the job cap only decides when the whole shard is killed with every test passing.
The failure this PR is preventing is a green shard killed by load variance: main's 8-shard runs on 2026-09-17 varied 24-46 min for the same split (run 35279898889), a 1.9x spread from runner contention alone.
At 4 shards the measured POSIX worst is 29 min (run 35299836749); a 50 cap leaves 1.7x, a 60 cap 2.1x against that spread. 60 is the operator's explicit ask ("增大timeout") for this reason, not symmetry with Windows.
The deleted "cap stays 50" pin argued that raising the wall could not clear a 33-min fixed cost; that premise is gone with the collection fix, so the pin is not inherited, it is retired with its reason.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Legitimate: gate-floor.md:166 still says "eight backend runners", and the same sentence also carries two other stale counts ("62,108 collected backend tests", "collection ~100s") that this PR's collection fix changes too.
Deferred rather than pushed: the PR is fully green on this head, and a one-line rationale-doc edit would re-run the 78-check matrix (~35 min of runners) for no behaviour change.
#11679 carries the deferred-finding label, an assignee, and Due: 2026-09-25, and rewrites the whole paragraph (all three stale numbers) instead of only the word this lane named.

@chenmingwei23
chenmingwei23 merged commit fddbaed into main Sep 18, 2026
106 of 109 checks passed
@chenmingwei23
chenmingwei23 deleted the perf/gate-pytest-asyncio-fixture-scan branch September 18, 2026 06:36
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 18, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…backend shards (kirodotdev#11636)

pytest-asyncio 0.20.3 walks every registered fixture on every collected
test function (O(tests x fixtures)): 87,244 scans x ~1,420 fixtures =
123.8M coroutine checks, 85% of collection under cProfile. Every CI shard
paid ~33 min of that before its first test ran, which is why kirodotdev#11528 went
to 8 shards. The rootdir conftest now re-runs the scan only after a
fixture registration (FixtureManager._register_fixture is the single
chokepoint): collection 478 s -> 68 s locally, identical 112,246 node ids.

CI: backend-test and backend-test-windows back to 4 shards each, both
caps 60 min (measured worst 4-shard shard: Linux 26 min, Windows 39 min).

Co-authored-by: Joe Guo <zejiangg@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants