Repository navigation
ci: shard backend test files before collection - #11718
Conversation
|
Intent: Divide ordinary Linux and Windows backend test files before their items are imported and collected, preserving the complete suite and existing acceptance gates. |
Design Review (Fable 5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Doubling to 16 backend jobs per PR risks the ~20 concurrent-runner ceiling; end-to-end gain for this revision is unmeasured. WatchThe deleted Suggestions
[DESIGN-REVIEWED] 3042877 |
First Principles Review (Fable 5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The 4→8 shard bump reverses the deleted "Why 4, not 8" pin's runner-pool decision on evidence the description itself calls unestablished. Not justified as shipped
What this change shipsInventory (8 items) — 7 justifiedIntent: make backend CI shards stop importing the whole suite before selecting their share, and fix the coverage attribution defect the partition exposes — a FIX (of measured cost, documented at
[FIRST-PRINCIPLES-REVIEWED] 3042877 |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
UX Review (Fable 5) — ⏭️ skippedRevision |
|
|
|
|
|
|
The first completed Linux shard reports 28,566 passed, 67 skipped, 5 xfailed and one failure in 600.32 seconds. The sole failure is test_backend_cards.py::test_every_capability_set_is_classified_by_the_card, missing ACP_BACKENDS_CLIENT_META_SETTINGS and ACP_BACKENDS_TOOL_SEARCH_OVERLAY. It reproduces in a one-test local run without file sharding. The capability definitions, card implementation and assertion test are byte-identical to this PR base; they are untouched by this patch. PR #11728 already fixes exactly these two missing classifications, so this PR will not duplicate that unrelated fix. The other shards remain in progress; this one completed result is not a full-matrix performance claim. |
|
5af39fb to
754fdc1
Compare
|
|
754fdc1 to
7e0dbcc
Compare
7e0dbcc to
e96a130
Compare
e96a130 to
3042877
Compare
|
|
|
iamwhatever
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: chore / build tooling (11 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: CI plumbing only -- replaces pytest-split duration groups with whole-file hash shards assigned before import (scripts/ci_file_shards.py plugin), takes Linux/Windows shard count 4 -> 8, adds two explicit --cov directories, and wires the conftest import-time-metric guard to fail its own collection report (with matching harness updates); no runtime code touched. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/common/testing-conventions.md.
Problem / Motivation
Each backend shard imports the whole test suite before selecting its share. More runners repeat that work instead of dividing it. File partitioning exposes a coverage measurement defect: exec-loaded source can run without being counted when its module name differs from its package name.
Why it matters
PR authors wait for tests their shard never runs. Divide that work before import, and measure the code that actually runs without weakening any gate.
What changed (motivation → approach → change)
The CI workflow (
.github/workflows/ci.yml) loadsscripts/ci_file_shards.py. A root-relative path hash gives each file one owner before import. Pytest retains discovery, platform ignores and explicit-target semantics. Eight shards divide the ordinary Linux and Windows suites; reduced-scope targets use the same owners, while leaf repeats remain unsharded. Hashing does not balance time, and one large file stays together. Extra shards use more runner slots and repeat setup.Linux keeps
--cov=kiro_crew --cov=sage_liband adds directory selectors only for AWS Control crew packaging and Sage tests. Those 63 Python files already appear in the original coverage report; the bounded roots fix synthetic-module and app-local import attribution without bringing standalone scripts, container code or vendored libraries into scope.setup.cfgretains branch coverage, relative paths, aliases and omit rules. Coverage upload, combine and thresholds stay intact. The sole baseline deletion is the Sage fixture, measured at 15/17 lines in CI. Compiled variants share the original filename; their contribution is not proof every counted line ran in an unmodified variant.Root
conftest.pyfails the offending module's collection report when its import emits a metric. Telemetry remains off and attribution/reset stay intact. The annotation guard recognizes both pytest invocation spellings. Tests and the owning CI/testing docs cover file ownership, import isolation, all-worker enforcement and coverage union. No Fleet, credential, sandbox or macOS changes; no five-minute guarantee.Backwards compatibility
Compatible: normal pytest commands and macOS retain selection behavior. Only the two ordinary backend matrices opt into file sharding. Their configured suite remains intact. Invalid flags, owned import errors, test failures and empty shards remain nonzero. Import-time metric emission was already forbidden; every detecting worker enforces it directly.
Tests
Manual verification
The real packaging subset ran 499 passed / 1 existing skip, both unsharded and partitioned into four filesets. Its exact 500-node union had no duplicates; both coverage results measured 1,677/1,844 builder statements (90.94%) with identical branch arcs. All 276 lines missing from the failing CI measurement were recovered. The Sage directory ran 810 passing tests; its fixture measured 15/17 with either import order.
Eight-shard file inventory (enumeration, not full pytest collection): 2,763 files distributed 336 / 356 / 337 / 356 / 338 / 358 / 321 / 361. Reduced selections are all nonempty: Linux 567 files, Windows 555. Executable regressions separately validate pytest's selection semantics.
The preceding four-shard run passed all eight backend jobs and E2E. Linux test steps took 14:39 / 18:31 / 18:40 / 16:50; Windows took 22:45 / 21:26 / 19:02 / 21:12. Coverage Gate failed on the measurement defect described above. Those timings are comparison evidence, not acceptance for this revision. The new eight-shard run must establish queue time, job duration, coverage and actual end-to-end improvement. Buffered log timestamps are not collection measurements.
Runner migrations remain separate from this change. Existing namespace jobs and macOS jobs stay untouched.
Rollback: restore each ordinary matrix and SHARD_COUNT to 4; replace
-p scripts.ci_file_shardsand its flag pair with--splits/--groupin the three matrix invocations. Keep the all-worker import guard and accurate coverage selector. No infrastructure change is needed.Related Issues
no linked issue: maintainer-requested CI performance change.
Pattern harvest
Rule candidate: lint
Pattern: shard before importing test files; coverage must not depend on the import name that first reaches a source filename.
Checklist