Skip to content

ci: shard backend test files before collection - #11718

Merged
iamwhatever merged 1 commit into
mainfrom
ci/file-level-sharding
Sep 18, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
ci/file-level-sharding

Conversation

@bolichen97

@bolichen97 bolichen97 commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Each backend shard imports the whole test suite before selecting its share. More runners repeat that work instead of dividing it. File partitioning exposes a coverage measurement defect: exec-loaded source can run without being counted when its module name differs from its package name.

Why it matters

PR authors wait for tests their shard never runs. Divide that work before import, and measure the code that actually runs without weakening any gate.

What changed (motivation → approach → change)

Before After
🟥 Collect all items, then split them 🟩 Assign files before collection
🟨 Four Linux and four Windows jobs 🟩 Eight file shards per OS
🟨 Import-order-sensitive measurement 🟩 Package scope plus two bounded roots
🟨 Import-metric assertion in one test 🟩 Offending collection fails on every worker
🟦 Runner routing and timeout budgets 🟦 Kept

The CI workflow (.github/workflows/ci.yml) loads scripts/ci_file_shards.py. A root-relative path hash gives each file one owner before import. Pytest retains discovery, platform ignores and explicit-target semantics. Eight shards divide the ordinary Linux and Windows suites; reduced-scope targets use the same owners, while leaf repeats remain unsharded. Hashing does not balance time, and one large file stays together. Extra shards use more runner slots and repeat setup.

Linux keeps --cov=kiro_crew --cov=sage_lib and adds directory selectors only for AWS Control crew packaging and Sage tests. Those 63 Python files already appear in the original coverage report; the bounded roots fix synthetic-module and app-local import attribution without bringing standalone scripts, container code or vendored libraries into scope. setup.cfg retains branch coverage, relative paths, aliases and omit rules. Coverage upload, combine and thresholds stay intact. The sole baseline deletion is the Sage fixture, measured at 15/17 lines in CI. Compiled variants share the original filename; their contribution is not proof every counted line ran in an unmodified variant.

Root conftest.py fails the offending module's collection report when its import emits a metric. Telemetry remains off and attribution/reset stay intact. The annotation guard recognizes both pytest invocation spellings. Tests and the owning CI/testing docs cover file ownership, import isolation, all-worker enforcement and coverage union. No Fleet, credential, sandbox or macOS changes; no five-minute guarantee.

Backwards compatibility

Compatible: normal pytest commands and macOS retain selection behavior. Only the two ordinary backend matrices opt into file sharding. Their configured suite remains intact. Invalid flags, owned import errors, test failures and empty shards remain nonzero. Import-time metric emission was already forbidden; every detecting worker enforces it directly.

Tests

  • Executable eight-shard regression preserves 26 files / 62 items, including parameters and ignores, with no duplicate or missing item.
  • Reduced-scope nonempty-bucket regression reads the workflow shard count. Both matrices require count 8 and groups 1 through 8. Negative controls reject count/list mismatches.
  • Unassigned import failures remain unimported; owned failures and malformed options still fail. Two xdist workers retain identical collection and loadgroup behavior.
  • An injected import-time metric fails serial and xdist collection without the assertion file. Old-guard negative controls fail.
  • Coverage regressions include synthetic exec names, mutations and both Sage import spellings. Combined branch arcs equal the unsharded result, with nonzero expected lines. Nested runs isolate their data files and preserve every staged shard.
  • Package-only negative control loses expected builder lines; whole-directory negative control admits an out-of-scope script. The bounded selectors pass both checks. Mixed-separator coverage keys are compared by normalized identity and queried by their exact stored spelling; missing or duplicate identities fail.
  • Parent verification: 231 related tests passed under coverage, including both workflow-contract files; the two affected files also passed all 30 tests with two xdist workers and coverage enabled. Native filename keys and nested coverage startup are checked without dropping line or branch assertions. Compile and style checks passed. Native Windows and whole-suite coverage require remote CI.

Manual verification

The real packaging subset ran 499 passed / 1 existing skip, both unsharded and partitioned into four filesets. Its exact 500-node union had no duplicates; both coverage results measured 1,677/1,844 builder statements (90.94%) with identical branch arcs. All 276 lines missing from the failing CI measurement were recovered. The Sage directory ran 810 passing tests; its fixture measured 15/17 with either import order.

Eight-shard file inventory (enumeration, not full pytest collection): 2,763 files distributed 336 / 356 / 337 / 356 / 338 / 358 / 321 / 361. Reduced selections are all nonempty: Linux 567 files, Windows 555. Executable regressions separately validate pytest's selection semantics.

The preceding four-shard run passed all eight backend jobs and E2E. Linux test steps took 14:39 / 18:31 / 18:40 / 16:50; Windows took 22:45 / 21:26 / 19:02 / 21:12. Coverage Gate failed on the measurement defect described above. Those timings are comparison evidence, not acceptance for this revision. The new eight-shard run must establish queue time, job duration, coverage and actual end-to-end improvement. Buffered log timestamps are not collection measurements.

Runner migrations remain separate from this change. Existing namespace jobs and macOS jobs stay untouched.

Rollback: restore each ordinary matrix and SHARD_COUNT to 4; replace -p scripts.ci_file_shards and its flag pair with --splits / --group in the three matrix invocations. Keep the all-worker import guard and accurate coverage selector. No infrastructure change is needed.

Related Issues

no linked issue: maintainer-requested CI performance change.

Pattern harvest

Rule candidate: lint
Pattern: shard before importing test files; coverage must not depend on the import name that first reaches a source filename.

Checklist

  • One Conventional Commits commit
  • Related tests pass and new regression tests added
  • Self-review completed; project style followed
  • Owning CI documentation updated
  • No secrets, credentials or internal references in the diff

@bolichen97
bolichen97 requested review from a team and buluoray September 18, 2026 07:02
@bolichen97

Copy link
Copy Markdown
Contributor Author

Intent: Divide ordinary Linux and Windows backend test files before their items are imported and collected, preserving the complete suite and existing acceptance gates.
Not a goal: Changing runner infrastructure, shard counts, timeouts, permissions, namespace jobs, macOS, leaf-repeat behavior, or promising five-minute full CI.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 3042877d106c147ccca7e4e375d94a5c1b1355eb — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Doubling to 16 backend jobs per PR risks the ~20 concurrent-runner ceiling; end-to-end gain for this revision is unmeasured.

Watch

The deleted ci.yml comment recorded why 4-not-8 was chosen ("at peak they queue the runner pool for every other PR — mind the ~20 concurrent-job ceiling for public repos"). This PR removes the collection cost that comment cited, but the queueing half is untouched: 8+8 backend jobs plus frontend/E2E/macOS lanes can exceed the ceiling, serializing jobs so a full-matrix PR — and its siblings — gets slower end-to-end despite smaller shards. The description concedes acceptance is pending: "The new eight-shard run must establish queue time, job duration, coverage and actual end-to-end improvement."
Clears when: an eight-shard run on this revision shows end-to-end wall clock and queue time at or better than the linked four-shard baseline, ideally alongside a concurrent sibling PR.

Suggestions

  • If the eight-shard run shows queueing, keep the plugin and drop only the count back to 4–6 per OS — the rollback section already treats count and mechanism as independent knobs; make that the first lever, not the full --splits revert.

[DESIGN-REVIEWED] 3042877

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 3042877d106c147ccca7e4e375d94a5c1b1355eb — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The 4→8 shard bump reverses the deleted "Why 4, not 8" pin's runner-pool decision on evidence the description itself calls unestablished.

Not justified as shipped

  • Item 2 — inherited (pin reversal, unverified premise): the deleted pin decided against 8 ("8 shards would mostly buy 8 x setup … at peak they queue the runner pool for every other PR — mind the ~20 concurrent-job ceiling"; Windows: "4 shards halves the Windows runners a full-matrix PR holds at peak"). File sharding answers the pin's collection half — collection now divides — but not the pool-pressure half, which ci.yml:2162 still cites for the frontend matrix, and the description concedes "The new eight-shard run must establish queue time, job duration, coverage and actual end-to-end improvement."
    Subtraction: shrink group: and SHARD_COUNT back to 4 in both matrices; the plugin works at 4, and the documented rollback already isolates the count from the mechanism.
    Clears when: an eight-shard CI run measured end-to-end (queue + wall) faster than four-shard run 35323920071, or the count is returned to 4.

What this change ships

Inventory (8 items) — 7 justified

Intent: make backend CI shards stop importing the whole suite before selecting their share, and fix the coverage attribution defect the partition exposes — a FIX (of measured cost, documented at testing-conventions.md: 14,237/56,946 items still paying the 744 MiB unsharded floor).

  1. Linux/Windows backend shards own whole files before import via new scripts/ci_file_shards.py, replacing pytest-split there — justified
  2. Shard count doubles from 4 to 8 per OS — inherited: reverses the deleted "Why 4, not 8" and Windows peak-runner pins on timings the description says are not yet established
  3. An import-time metric emission now fails that module's collection on every worker and every run, not just one test's assertion — justified
  4. Linux coverage adds two bounded directory selectors so exec-loaded and app-local-named source is counted regardless of import order — justified
  5. Coverage baseline drops the Sage fixtures 0.0% entry, now measured — justified
  6. CI invokes python -m pytest; the annotation-matcher guard recognizes both spellings — rides along; derived (makes scripts.ci_file_shards importable)
  7. File-shard flags refuse to combine with --splits — justified
  8. Owning CI and testing docs rewritten in the same commit — justified

[FIRST-PRINCIPLES-REVIEWED] 3042877

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 3042877d106c147ccca7e4e375d94a5c1b1355eb and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 3042877

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3042877d106c147ccca7e4e375d94a5c1b1355eb: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 3042877d106c147ccca7e4e375d94a5c1b1355eb — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3042877

Verdict parsed from the review's SHA-scoped output markers for commit 3042877d106c147ccca7e4e375d94a5c1b1355eb.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3042877d106c147ccca7e4e375d94a5c1b1355eb: <one-sentence reason>

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Fable 5) — ⏭️ skipped

Revision 3042877d106c147ccca7e4e375d94a5c1b1355eb touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@bolichen97

bolichen97 commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=34924f3a17c5

fixed: 5af39fb98a updates all four named sites in testing-conventions.md and conftest.py. The local reproduction uses scripts.ci_file_shards with --file-shards/--file-shard.
References to pytest-split remain only where still applicable: macOS, optional duration recording, and historical measurements. The owning spec also documents per-worker metric-guard enforcement. Documentation lint and related tests pass.

@bolichen97

bolichen97 commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=cd92c409e0e5

rebutted: the production reduced selector at the repaired revision emits 567 files, assigned 152/137/140/138 to the four shards. It selects a broad cross-surface set, not the few files changed by a frontend PR; the separate leaf-repeat path exits before sharding.
Exit 5 is deliberately retained. If a future selector change produces an empty partition, that change must reconcile its matrix rather than silently turn missing execution green. No currently reachable empty reduced shard was found. The PR adds tests for disjoint reduced targets and nonzero empty selections; no success-on-empty fallback is added.

@bolichen97

bolichen97 commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=8e60acc2cb2c

fixed: 5af39fb98a makes the detecting worker fail the actual module collection report. Enforcement no longer depends on the shard containing test_host_isolation_floor.py.
The unconditional telemetry-off pin, per-module attribution and recorder reset remain intact. Existing collection failures retain their original diagnostic.
Negative controls: planted emissions on a shard without the assertion test returned 0 with the old guard. The repaired guard returns 2 serially and 1 through the xdist controller, without INTERNALERROR. Clean runtime emission/reset passes. Parent verification: 199 focused tests plus compile/style/docs checks passed; focused independent review found no new Critical/High.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=bb08d20688da

rebutted: the current GitHub pulls/11718/files endpoint reports exactly eight files and no website/ paths. git diff b05c62d 5af39fb reports the same eight files; the entire website/ diff is empty.
The changed files are ci.yml, conftest.py, the two owning docs, ci_file_shards.py and three test files. The reported useSilentLoadWatch.ts/frontend cluster does not belong to this PR. Rebasing or changing code to remove nonexistent hunks would not fix anything.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=24049ab08f4b

rebutted: measured the current ci-surface-tests.py --surface backend --summary output with the shipped file_shard() function. It returns 567 files, distributed 152/137/140/138 across shards 1/2/3/4. All four shards own files.
This is the broad cross-surface selector, independent of the handful of files a frontend PR changes. Leaf-test repeats exit before this branch. The requested evidence is satisfied; retaining nonzero exit on a future broken/empty selection is intentional.

@bolichen97

Copy link
Copy Markdown
Contributor Author

The first completed Linux shard reports 28,566 passed, 67 skipped, 5 xfailed and one failure in 600.32 seconds. The sole failure is test_backend_cards.py::test_every_capability_set_is_classified_by_the_card, missing ACP_BACKENDS_CLIENT_META_SETTINGS and ACP_BACKENDS_TOOL_SEARCH_OVERLAY.

It reproduces in a one-test local run without file sharding. The capability definitions, card implementation and assertion test are byte-identical to this PR base; they are untouched by this patch. PR #11728 already fixes exactly these two missing classifications, so this PR will not duplicate that unrelated fix. The other shards remain in progress; this one completed result is not a full-matrix performance claim.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=88697af415fd

rebutted by completed-run evidence: CI run 35320236770 completed all eight test steps without a job timeout. Linux shard test-step durations were 10:09, 19:02, 18:47, 18:13; Windows were 24:48, 23:26, 22:37, 23:56. The slowest retained over 35 minutes of the unchanged 60-minute budget.
This satisfies the requested budget-headroom check, not full correctness or a five-minute promise. Linux1/Windows1 failed the pre-existing capability-classification test fixed in merged PR11728; Linux2/Windows2 exposed a pytest-invocation detector that is being corrected in this PR; Windows4 hit an approval-test timing failure that still needs disposition. Coverage remains blocked by the failures and must be green on the next revision. No timeouts were raised and no tests skipped to obtain these timings.

@bolichen97
bolichen97 force-pushed the ci/file-level-sharding branch from 5af39fb to 754fdc1 Compare September 18, 2026 08:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=d9b5853e655b

Disposition: rebutted with current-head measurement. All four Linux and four Windows backend jobs passed in run 35323920071 on this exact head.
Test-step times, shard order 1/2/3/4: Linux 14:39 / 18:31 / 18:40 / 16:50; Windows 22:45 / 21:26 / 19:02 / 21:12. Whole jobs also completed within the unchanged 60-minute cap (longest Windows job 23:42).
Evidence: https://github.com/kirodotdev/KiroCrew/actions/runs/35323920071 . The timing watch is answered, but this does NOT claim complete CI acceptance: the downstream Coverage Gate is red and its measurement defect is under investigation. No five-minute guarantee or timeout increase.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=c7c2032f3f56

Disposition: rebutted. Reduced backend scope is not the files changed by a frontend PR. scripts/ci-surface-tests.py collect("backend") scans all configured backend roots and selects every cross-surface test, independent of the PR diff.
The current-head Opus review independently verified hundreds of selected files (500+ under test/ alone) and rejected the single-file premise. The prior measured distribution was 567 files across four nonempty owners (152/137/140/138), not an arbitrarily small changed-file set.
Keep pytest exit 5 as a failure: suppressing it would mask a broken or unexpectedly emptied selection. No fallback to the full suite, no skipped guard, and no new tolerance is introduced.

@bolichen97
bolichen97 force-pushed the ci/file-level-sharding branch from 754fdc1 to 7e0dbcc Compare September 18, 2026 10:17
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@bolichen97
bolichen97 force-pushed the ci/file-level-sharding branch from 7e0dbcc to e96a130 Compare September 18, 2026 10:49
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 18, 2026
@bolichen97
bolichen97 force-pushed the ci/file-level-sharding branch from e96a130 to 3042877 Compare September 18, 2026 11:42
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=cda0dcf6bcf1

Disposition: rebutted as a present change request; retain the existing count-only rollback. The current eight-shard CI passed in 18:36 versus 25:05 for the linked four-shard baseline, with backend dispatch-to-start upper bound 73s versus 76s (both include dependency waits).
There is no observed queue regression in this run requiring a count reduction. The PR rollback already separates matrix.group/SHARD_COUNT from the pre-import plugin and coverage repair: a later demonstrated pool regression can reduce the count without undoing either fix.
Evidence: https://github.com/kirodotdev/KiroCrew/actions/runs/35340954833 . Keep the user-requested eight shards now; this is not a promise that every future load fits the pool.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=42b3eaafdbdb

Disposition: rebutted with the requested eight-shard measurement. Current-head CI35340954833 is success, including all sixteen backend shards, E2E and Coverage Gate. Workflow creation to last CI job completion is 18:36 against 25:05 for four-shard CI35323920071.
Backend completion since workflow dispatch improves Linux 20:39 -> 14:02 and Windows 24:55 -> 15:54. Backend startup upper bound is 73s versus 76s, including dependency waiting. These are queue-inclusive observed wall times, not inferred collection timings.
Six sibling PR CI workflows were created during the run. This is evidence for retaining the explicitly requested eight shards, not proof of peak capacity or a controlled same-source comparison. Count rollback remains independent from the plugin.
Evidence: https://github.com/kirodotdev/KiroCrew/actions/runs/35340954833 and https://github.com/kirodotdev/KiroCrew/actions/runs/35323920071 . Separate macOS desktop build completion is not included in these CI timings.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=d68a63689dd8

Disposition: rebutted with completed current-head evidence. CI run 35340954833 passed every test and Coverage Gate: creation-to-last-job completion was 18:36, versus 25:05 for the four-shard run 35323920071 (whose tests passed but coverage failed).
Maximum backend dispatch-to-job-start delay, including dependency waits rather than claiming pure queue time, was 73s versus 76s. All sixteen backend jobs started within 57–73s of dispatch; the four-shard baseline was 72–76s.
Linux slowest test step improved 18:40 -> 12:31; Windows 22:45 -> 14:04. Six other PR CI runs were created during this run (35340997498, 35341048789, 35341194671, 35341769536, 35341874584, 35341944516), so the observation was not an isolated workflow.
Evidence: https://github.com/kirodotdev/KiroCrew/actions/runs/35340954833 and https://github.com/kirodotdev/KiroCrew/actions/runs/35323920071 . This proves this observed run improved, not a controlled same-source benchmark or a guarantee about peak fleet capacity. The separate macOS desktop build is still pending; no five-minute or whole-PR completion claim.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 18, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 18, 2026 13:05

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: chore / build tooling (11 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: CI plumbing only -- replaces pytest-split duration groups with whole-file hash shards assigned before import (scripts/ci_file_shards.py plugin), takes Linux/Windows shard count 4 -> 8, adds two explicit --cov directories, and wires the conftest import-time-metric guard to fail its own collection report (with matching harness updates); no runtime code touched. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/common/testing-conventions.md.

@iamwhatever
iamwhatever merged commit ba60bfe into main Sep 18, 2026
88 checks passed
@iamwhatever
iamwhatever deleted the ci/file-level-sharding branch September 18, 2026 13:06
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 18, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants