Skip to content

ci: run each linux backend shard as single-process pytest parts - #39738

Closed
Zedmor wants to merge 5 commits into
mainfrom
ci/linux-shards-run-sharded-suite
Closed

Zedmor wants to merge 5 commits into
mainfrom
ci/linux-shards-run-sharded-suite

Conversation

@Zedmor

@Zedmor Zedmor commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Linux backend shards use less compute and finish sooner, because each pytest process collects only the files it runs.

Each Linux backend shard runs one pytest -n $WORKERS. Every xdist worker collects the whole shard before it runs a test: 123-234 s a worker, six workers a shard, on main run 44620769265. A worker that holds a whole shard's collection costs about 1.5-1.8 GiB, and that memory, not the 8 cores, is what holds the count at 6 on the 15 GiB fleet runner (the comment in the Run tests step records 15,050 of 15,738 MiB at 8 workers).

scripts/run_sharded_suite.py already runs a suite as many single-process pytest runs that each collect only their own files. CI does not use it (#39702).

Why it matters

Linux backend shard job-minutes were a median 175.5 per full-suite run after the latest merges (194 runs), against a target of 127 or less, and the slowest Linux shard a median 26.6 minutes against a target of 15 (#39702). Collection repeated in every worker and two idle cores per shard are a fixed cost of every full-suite run on every pull request.

Not a goal

  • Windows and macOS lanes: untouched. ci: run the Windows backend shards with 6 xdist workers #39666 edits only the Windows worker-count step; this diff stays inside the Linux backend-test job, scripts, tests and docs.
  • The reduced and leaf arms of the Linux job: byte-identical.
  • A new sub-shard planner or a duration-balanced part split: the parts are scripts/ci_file_shards.py's path-hash split.
  • Changing which tests a shard runs, the shard count, the pin file, the prewarm step's behaviour, Coverage Combine or Coverage Gate.
  • Proving the gain: that is the measurement plan below. This PR stays a draft until it is done.

What changed (motivation → approach → change)

Goal: one collection per file instead of one per worker, and a process small enough that one fits on every core.

Approach: keep the outer split exactly as it is (8 shards, same hash, same pin file) and split one shard's files again with the same plugin. The script lists the files the outer shard owns and hands the list to every part as --file-shard-targets; ci_file_shards then gives each file to one of the parts by path hash. The list only narrows discovery, so pytest's own patterns and conftest ignores still decide what is collected, and the union of the parts is the shard.

Change:

  • scripts/run_sharded_suite.py: --outer-shards/--outer-shard/--outer-pins (run only the files CI shard K of N owns), --coverage-file (own COVERAGE_FILE per part, combined when every part passed), --run-timeout (one deadline for the whole run: kill and report what still runs, start nothing further) and --part-timeout (the same, from a part's own start), --print-logs (copy each part's log to stdout, folded on GitHub Actions: when the part ends, once when its log has not grown for six minutes, and when a stop ends it). A line a minute names the running parts. A part that exits 0 without a JUnit file counting a test is red.
  • .github/workflows/ci.yml, Linux Run tests, full-suite arm only: python scripts/run_sharded_suite.py --jobs "$JOBS" --shards "$PARTS" --outer-shards "$SHARD_COUNT" --outer-shard <g> --outer-pins scripts/ci_file_shard_pins.txt --out "$RUNNER_TEMP/sharded-suite" --print-logs --run-timeout "$RUN_SECS" --coverage-file .coverage -- <the same pytest arguments>, where RUN_SECS is the job's timeout-minutes less 10 minutes. A new linux-part-logs-<g> artifact (always()) keeps every part's log and JUnit file.
  • test/test_cron_auto_pause_limit.py: imports chat_folders_paths before its tests patch kiro_crew.sel.sel (the one failure of the first CI run, below).
  • Docs: the backend-test row and "Backend file sharding" in docs/ci/ci-and-reviews.md; "Shard balance" and the runner paragraph in docs/system-specs/common/testing-conventions.md.

The seven design questions

  1. Coverage. Each part gets COVERAGE_FILE=<out>/coverage-<i>/.coverage, its own directory, so no two processes write or combine the same file (pytest-cov at -n0 combines everything beside its data file). After the last part the script runs coverage combine --data-file=.coverage <every part's files> from the repository root, only when every part passed. A part that ran tests and left no data makes the run red, a red run leaves no .coverage (a file from an earlier run is removed first), and "Stage shard coverage data" still requires exactly that file, so Coverage Combine and Coverage Gate are unchanged and fail closed. Local result: same 4,773 measured files; total for shard 1 44.010-44.018 % on the old command and 44.009-44.015 % as parts (numbers below).
  2. Verdict. The script exits non-zero when any part exits with a status other than 0 or 5 (failed test, collection error, internal error, death by signal), exits 0 without a JUnit file that counts a test (reported as 125), or is still running at the run's deadline (group killed, reported as 124; a part still queued then is not started, reported as 126). Every part is started and waited for; statuses are collected per part and printed, never replaced. Exit 5 (no tests collected) counts as passed for one part, because the split is by hash and a part may own no file with a selected test; its files belong to other parts. It is not a way to go green by accident: when every part exits 5 the run exits 5, and node-id equality with the old command is checked below. With 48 parts over about 1,020 files no part was empty on any shard in a replay of run 44620769265.
  3. Scope arms. The issue names "each Linux backend shard" and the pin file; only the full-suite arm takes the pins and coverage, and it is the arm whose every worker collects a whole shard. The reduced and leaf arms are byte-identical (git diff shows no changed line inside them).
  4. xdist_group. 283 literal group names in the tree, 214 of them tree_scan_* over 219 files. Only 4 tree_scan_* groups span files (2, 2, 3 and 7 files); most name one file, and a one-file group needs no pin. All multi-file groups together pin 79 files in the whole tree; the largest set that lands in one part of one outer shard is 5 files. So grouping does not skew part balance or memory. At -n0 a part is serial, so a group's tests still run one at a time, as on one xdist worker.
  5. Plugins and artifacts. ci_pytest_progress names its file <worker>-<pid>-<uuid>.jsonl and opens it exclusively; at -n0 each part writes one controller-<pid>-<uuid>.jsonl into the same $RUNNER_TEMP/pytest-progress, so the upload step and scripts/ci_file_shard_pins.py (which globs *.jsonl and reads phase_end records) are unchanged; locally 48 parts left 48 files whose collection_end counts sum to the shard's 39,165 tests. ci_faulthandler_budget and ci_main_red_skip are per process and read only options and the environment. The outer pin file is read once by the script; the parts read the script's own group-pins.txt. JUnit, log and coverage paths carry the part index.
  6. Process count and memory. Same shape as WORKERS: memory after 2 GiB for the host, capped at the cores, but 1 GiB a process instead of 2: JOBS = min(nproc, (MemTotal - 2048 MiB) / 1024), 8 on the fleet runner and 4 on a 4-core hosted one. Measured footprint (summed RSS of the process tree, sampled every 5 s, shard 1): six xdist workers 9.1-9.4 GiB; six parts at a time 3.5-3.8 GiB; eight parts at a time 4.5-4.7 GiB, about 0.6 GiB a process. The peak-memory sampler and report step are unchanged. This count is the draft's candidate, not a decision: the issue says N is picked from the draft's peak memory, so the first runs decide whether 8 stays. Bounded waits: --run-timeout is the job's own budget less 10 minutes, counted from the start of the step and shared by every part, so a part that starts late has no later deadline than one that starts first; the job's timeout-minutes is unchanged. A dead process is a non-zero status at once. First CI run at 8: peak 4,265-5,898 MiB of 15,738 MiB.
  7. Inside a part. One process, -n0. The issue asks for "N processes instead of one pytest -n call" where "each process collects only its own files"; an xdist controller plus workers inside a part would collect the part twice or more and add a process per part.

Local numbers (shard 1 of 8, CI flags, same tree)

Development host: 32 vCPU, 123 GiB, load average 25-49 from other work during every run, so the ratios mean more than the absolutes. Old = pytest -n 6 with the step's flags; new = the script. Each run started from an empty bytecode cache and ran the prewarm first.

run cores command wall summed collection longest collection peak RSS (tree) cases passed / skipped / failed
old1 32 -n 6 723 s 669 s 112 s 9,112 MiB 39,165 39,068 / 97 / 0
old2 32 -n 6 759 s 629 s 105 s not sampled 39,165 39,067 / 97 / 1
new1 32 6 at a time, 48 parts 687 s 462 s 15 s 3,500 MiB 39,165 39,067 / 97 / 1
new2 32 6 at a time, 48 parts 732 s 476 s 16 s 3,511 MiB 39,165 39,066 / 97 / 2
oldP 8 (pinned) -n 6 836 s 612 s 102 s 9,269 MiB 39,162 39,065 / 97 / 0
oldP2 8 (pinned) -n 6 743 s 615 s 103 s 9,389 MiB 39,162 39,065 / 97 / 0
newP8 8 (pinned) 8 at a time, 48 parts 595 s 528 s 22 s 4,653 MiB 39,162 39,065 / 97 / 0
newP8b 8 (pinned) 8 at a time, 48 parts 525 s 477 s 20 s 4,524 MiB 39,162 39,065 / 97 / 0
newP6 8 (pinned) 6 at a time, 36 parts 732 s 417 s 17 s 3,779 MiB 39,162 39,065 / 97 / 0
  • Same count of processes (6): 0.96x wall on 32 cores (means 710 s against 741 s; the two ranges overlap), 0.93x on 8 pinned cores (one run). Not a clear gain by itself: every part pays its own start-up (about 12 s here), which eats most of what the single collection saves.
  • The count memory now allows (8 on 8 cores): 0.71x wall (means 560 s against 790 s; spread 0.63x to 0.80x across the four pairings), at half the peak memory. This is the configuration the workflow uses.
  • Node ids: identical sets in every run after dropping the @<group> suffix --dist loadgroup appends to a node id (39,165; 39,162 in the pinned runs, which deselect the test named next). Collect-only on two more shards: shard 4, 36,062 ids, and shard 7, 36,958 ids, identical between the old command and 48 parts.
  • The one test that failed: test/test_md_notebook_real_sandbox.py::test_the_move_waits_for_a_sandbox_that_masks_only_the_old_dir ([leader-exited], once also [undumpable]) failed in old2, new1 and new2 and passed in old1. It scans the host's whole process table and answers None when the table keeps changing; this host runs other agents' work. It fails on the old command too, so it is host noise, and the pinned runs deselect it in both modes to get green runs for the coverage check.
  • Coverage (one .coverage per run; new1 and new2 were red, so theirs were combined by hand from the parts for this table): 4,773 measured files in every run. Executed lines: old 562,541 / 562,559 / 562,627 / 562,650; new 562,510 / 562,544 / 562,552 / 562,570 / 562,577. coverage report --format=total: old 44.010 / 44.017 / 44.018 %, new 44.009 / 44.013 / 44.015 %. Sample files (hooks.py, older_sandboxes.py, platform_compat.py, sandbox.py): identical statement, miss, branch and partial counts. Two old runs differ from each other in 45-47 files; old and new differ in 77-92 files by a net 11 to -117 lines of 562,600: lines whose execution depends on which tests share a process. Whether any per-file floor in Coverage Gate notices is for the CI runs to show.

First CI run (44661146070, head 6028f59, one run: not a verdict)

All eight Linux shards took the full-suite arm: "pytest processes: 8 at a time, 48 parts". Main's column is run 44620769265, an earlier commit with an earlier pin file, so the shards do not hold the same files; the plan below compares against main runs of the same day.

shard Run tests here main longest collection here main (each of 6 workers) peak memory here collected here result
1 868 s 955 s 33 s 170 s 4,823 MiB 39,241 green
2 946 s 987 s 36 s 176 s 4,323 MiB 35,354 green
3 1,115 s 1,325 s 36 s 215 s 4,424 MiB 36,116 green
4 1,114 s 1,300 s 34 s 217 s 4,522 MiB 36,101 green
5 838 s 1,335 s 28 s 234 s 4,401 MiB 38,472 green
6 911 s 983 s 46 s 201 s 4,265 MiB 35,561 green
7 859 s 1,386 s 26 s 233 s 5,898 MiB 36,973 red: 15 tests in part 20
8 863 s 835 s 28 s 135 s 5,558 MiB 36,862 green
sum / max 7,514 s / 1,115 s 9,106 s / 1,386 s max 5,898 of 15,738 MiB
  • Summed Run tests 0.83x of that main run, slowest shard 0.80x. Summed test seconds rose from 42,446 to 45,234 (+7 %): eight processes on eight cores slow each test a little, as the issue's first risk says.
  • The red shard is a defect this change exposed, and it is fixed here. test/test_cron_auto_pause_limit.py replaces kiro_crew.sel.sel for one test; add_job then imports kiro_crew.dashboard.chat_folders_paths for the first time, which binds sel by name, so the stand-in stays bound after the test. The next first import of the chat_folders facade in that process refuses a family whose owners disagree on sel, and 15 tests in test_power.py, test_slot_list_pass.py, test_subagent_undelivered_notice_restart.py and test_telegram_parity_approvals.py fail at that import. A worker that collected the whole shard had imported the module at collection, before any test ran; a part imports far less at collection. Reproduced locally with pytest -n0 test/test_cron_auto_pause_limit.py test/test_power.py (1 failed) and with the whole part (15 failed, 866 passed); with the import at the top of the test file both pass (89 passed; 881 passed, 1 skipped). It is deterministic for a given split, so it is a class the runs must keep watching: a different part count or pin file can put another such pair in one process.
  • Coverage Gate failed closed because a shard was red (backend-test=failure -- failing closed), and Coverage Combine was skipped; the seven green shards each produced their one .coverage.
  • Bundle Size Gate failed on first-load-2, 2.0 KB over its budget: main was red on it (Bundle Size Gate fails on main: first-load-2 is 1.9 KB over its ceiling #39726, fixed by ci: re-measure the first-load-2 bundle ceiling #39727); the branch is rebased past the fix.

Measurement plan (draft until done)

Runs: this PR's own full-suite run plus at least four more full runs of the same head. For each Linux shard on each run, next to main's same shard: Run tests step time, job time, collection seconds per process and summed (from the linux-pytest-progress-<g> artifact), peak memory (the "Report peak memory" step), tests collected and passed, and every failure with its test id.

Ship criteria. From the issue's Acceptance, quoted:

  • "The 3 draft runs and their table are in the PR body."
  • "On 3 sample shards, the set of test node ids run (passed plus skipped plus failed) equals the set the current single-process step runs on the same commit."
  • "Coverage Gate gives the same pass or fail as main on the same commit."
  • "CI green."

Plus:

  • collected and passed counts equal to main's for every shard on every run;
  • total coverage equal to main's within rounding, Coverage Combine and Coverage Gate green;
  • zero new failures across all runs (a test red here that is not red on main in the same window);
  • peak memory on every shard leaves the process count inside the runner (the issue: "Do not raise the process count past what the draft's peak memory allows"); if 8 does not hold, the count drops to the measured one and the runs are repeated;
  • Linux Run tests time summed over the shards lower than main's, and the slowest shard not slower.

If any criterion fails, the PR is fixed and re-measured, or closed with the numbers. One performance change this week passed on one run and then failed 41-45 % of the time on main (#39390), which is why one green run decides nothing here.

Known risks the runs must answer: timeouts when 8 processes share 8 cores (the issue's first risk); another test pair like the one above, where a module first imported in the middle of a test keeps a patched name; a repository-root residue check that now runs once per part, so a file one part's test leaves in the checkout while another part ends could fail that other part; the hosted 4-core fallback, where the count stays 4 and only the single collection is gained; and the prewarm step, which now only moves each file's one rewrite ahead of the parts.

Order: the issue says this lands after the change to 12 shards if that is approved. PARTS follows JOBS and the outer shard count comes from SHARD_COUNT, so nothing here names 8.

Backwards compatibility

Compatible: the script's existing options and defaults behave as before for a run that passes none of the new flags, except that a part exiting 0 with no JUnit result is now red; the reduced and leaf arms of the workflow are unchanged.

Removes nothing: the full-suite arm runs the same tests with the same plugins and coverage selectors through a different launcher; no file, option, artifact or check is deleted or renamed.

Tests

  • test/test_run_sharded_suite.py (56 tests): the outer shards own every file exactly once and honour the pins; the targets file is what ci_file_shards.read_targets reads; a part of an outer shard is narrowed to its files; half-named outer shards and a zero bound are refused. Verdict, each with real child processes: a part that dies by signal makes the run red while the others finish; the timing logic runs in the calling thread against an injected clock, with stand-in parts and no process, FIFO, thread or event: a part still running at its own bound is killed then and one that ends before it is not; a part that starts late still ends at the run's deadline; the bound is the earlier of the two (five cases); a part's bound counts from before its launch; a part whose turn comes after the deadline is not started (126); the running parts are named once a heartbeat; a part gone quiet has its log printed once while it still runs, and again only after it grew; a stop prints the log so far of every running part, before the parts are stopped. Real processes are started in one place: a module-scoped fixture runs two pytest parts and the script's coverage combine once, under a named 60 s run ceiling and a named 60 s combine ceiling inside the 300 s its consumers allow, and the tests that need what real parts leave read it from that fixture. The module carries one xdist_group, so every real child in it starts on one worker. There is no single shared child process: the children are different programs with different lifetimes (pytest parts that must run to their end, and a combine that reads what they wrote), so what is shared is the one run, not one process. A real part is no longer killed in a test of its own; the kill decision is tested against the clock with stand-in parts, and the signal itself goes through _signal_group, which the file's earlier stop tests exercise; a part that exits 0 without a result is red; an empty part (exit 5) is not red and a run of only empty parts is; each part writes its own coverage and the run combines them; a part that ran tests without coverage data is red and nothing is combined, even over stale files; a red part leaves no combined coverage; a combine that does not end is red; --print-logs copies every log. Two tests read the one nested run of real pytest parts over a temporary tree: the outer shard's tests run once each and a file of the other outer shard, which raises on import, is never collected; the coverage is one file with every part's lines. Files a test writes and then compares byte for byte are written as bytes (Windows runs these tests).
  • Workflow shape, updated deliberately: test_ci_pytest_progress.py (every pytest invocation carries the worker count, progress plugins and timeout; the full arm is the script with --jobs "$JOBS", a memory-and-cores rule, --run-timeout derived from the same expression as the job's timeout-minutes less a margin of at least 5 minutes, --print-logs, the part-log artifact under always(), and the coverage file the staging step requires), test_ci_file_shards.py (outer shard and pin file on Linux, no pins in the reduced and leaf arms), test_ci_rewrite_prewarm.py (prewarm and test step name the same shard and pin file), test_ci_ipv6_routing.py (the script invocation also deselects ipv6_required and real_venv), test_prepare_pr_profiles.py (the suite runner is not a pre-push static gate, with the reason).
  • Run locally on head 2f55227: the 177 test files that read ci.yml or the two docs, plus test_cron_auto_pause_limit.py and test_power.py -> 8533 passed, 77 skipped; test/test_run_sharded_suite.py 16 times at once on 4 pinned cores -> 16 x 56 passed, and with GITHUB_ACTIONS=true and without. Not run locally: Windows. Black, isort, flake8, subprocess-encoding, comment-history, brand, file-line-cap and docs-lint gates pass.

Manual verification

The local runs above, and the first CI run: the step ran under ci-shell as the unprivileged user on 8 cores and 15 GiB on all eight shards. Not verified: that a cancelled job's signal reaches the script through ci-shell (runuser then bash), so the log dump on a stop is proven only by the unit test; the deadline, the stall print and the artifact do not depend on it. Not verified on a runner: the run deadline and the stall print (no part has hung), and the hosted 4-core fallback.

Real-machine verification

Machine: own development host (EC2, AMD EPYC 9R14), Amazon Linux 2023, x86_64, 32 vCPU / 123 GiB, shared with other work (load average 25-49)
Before (main 49e045e): taskset -c 24-31 python -m pytest <PROGRESS plugins> -p scripts.ci_file_shards -q -n 6 --timeout=120 -m "not ipv6_required and not real_venv" --file-shards 8 --file-shard 1 --file-shard-pins scripts/ci_file_shard_pins.txt <--cov selectors> --cov-report= -> 39,065 passed, 97 skipped in 836 s and 743 s; six collections of 102-103 s; peak 9.3-9.4 GiB
After (the script of head 6028f59 on the same tree, less the three lines that clear an earlier run's coverage files; the host-noise test named above deselected in both commands): taskset -c 24-31 python scripts/run_sharded_suite.py --jobs 8 --shards 48 --outer-shards 8 --outer-shard 1 --outer-pins scripts/ci_file_shard_pins.txt --coverage-file .coverage --print-logs --part-timeout 3000 -- <the same pytest arguments> -> the same 39,065 passed, 97 skipped in 595 s and 525 s; 48 collections of at most 22 s; peak 4.5-4.7 GiB; one .coverage
Cleanup: own computer; the benchmark worktree is removed
CI runners: one run so far (table above), seven shards green and one red on the defect since fixed. Run 44666615747 on d301b89 ran no backend shard: AI Review Hold stopped the heavy jobs. Run 44669849461 on 6fc8b58: seven shards green (peak memory 4,437-6,300 MiB of 15,738; Run tests 845-1,146 s, 7,682 s summed), shard 7 red on three tests this PR had added, which asserted a plain log title where the script prints a ::group:: under GITHUB_ACTIONS; they are replaced in the third commit. Run 44674727926 on 9e67e6f: all eight Linux shards green, Coverage Combine and Coverage Gate green. Run tests 970-1,179 s (8,455 s summed), peak memory 4,318-6,897 MiB of 15,738. Backend coverage 94.30 %, the same as main's two latest green runs (44672595354, 44668256277); per-file floor 4,651 of 4,659 files at or above it with 8 baselined, identical to main's latest. That run was red on Windows shard 4 only: two tests this PR had added compared a log's bytes with text written in text mode (CRLF on Windows); fixed in the fourth commit. Three Linux runs so far sum to 7,514 / 7,682 / 8,455 s, a spread of about 12 %. Run 44682914304 on 8fa7065 (main merged in): the whole run is green, all eight Linux and all eight Windows shards, Coverage Combine and Coverage Gate, and every review lane. Run tests 935-1,164 s (8,158 s summed), peak memory 4,275-4,924 MiB, backend coverage 94.30 % with 4,653 of 4,661 files at or above the per-file floor and 8 baselined. Main's three latest green runs of the same morning (44681606779, 44675524626, 44672595354) sum to 8,504 / 9,526 / 9,411 s with slowest shards of 1,186 / 1,453 / 1,358 s; this PR's four runs sum to 7,514 / 7,682 / 8,455 / 8,158 s with slowest shards of 1,115 / 1,146 / 1,179 / 1,164 s. Means 7,952 s against 9,147 s, 0.87x, and the two ranges touch (8,455 against 8,504), so the gain on the runner is smaller than the 0.71x measured locally and is not yet separated from noise. Two runs have all eight Linux shards green; the plan asks for five.

Commits: the branch carries four commits (the change, the review fixes, the test reshape, the Windows and child-process fixes) and one merge of main, which resolved a conflict in the job table of docs/ci/ci-and-reviews.md with #39666; each was pushed without a force-push at the reviewer's request; whether to squash is the owner's call before this leaves draft. The merge is a squash either way. The remaining runs are the measurement plan.

Related Issues

Related: #39702

A closing keyword replaces Related only when the measurement plan passes.

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

AI-assisted, opened by Claude Code

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 8fa706577e07bee4d8328845a2f635e20cf18c4e touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 8fa706577e07bee4d8328845a2f635e20cf18c4e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: it is a CI-only change that a revert undoes, coverage and the verdict fail closed, and the reduced and leaf arms are untouched.

Notes

  • test_cron_auto_pause_limit.py breaking on the first run shows that a new mix of files in one process exposes latent cross-file pollution. The node-id equality check doesn't cover order-dependent failures, so expect more of these on the first several full runs before the PR leaves draft.
  • Six parts per process is a fixed per-part start-up cost picked without measurement. Let the measurement plan set the multiplier along with JOBS, rather than shipping it as a constant.

[DESIGN-REVIEWED] 8fa7065

Description read

[DESCRIPTION-READ] ab2f168b92d8a7b40994b8dbbea3c792ea015816b194049acea09f91a2762656

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 8fa706577e07bee4d8328845a2f635e20cf18c4e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Every item traces to the measured per-worker collection and memory cost in #39702. One flag ships with no caller.

Notes

  • Item 5 — zero consumers: --part-timeout has no caller. CI deliberately avoids it (self.assertNotIn("--part-timeout", options)), and grep finds it only in the script, its tests and docs. Subtraction: delete --part-timeout, Parts.timeout and the per-part half of bound(), so --run-timeout is the only bound.

What this change ships

Inventory (9 items) — 8 justified

Intent: make Linux full-suite backend shards cheaper and faster by having each pytest process collect only its own files. This is an ADDITION (CI efficiency).

  1. Linux full-suite shards run as single-process pytest parts through run_sharded_suite.py instead of one pytest -n — justified
  2. The number of processes at a time is sized by memory (1 GiB each after 2 GiB for the host), capped at the core count — justified
  3. --outer-shards/--outer-shard/--outer-pins limit the script to the files of one CI shard — justified
  4. --coverage-file gives each part its own coverage file and combines them only when every part passed — justified
  5. --part-timeout adds a per-part deadline — zero consumers
  6. --run-timeout derived from the job budget, plus the minute-by-minute line and the stall log dump, so a hung part gets named — justified
  7. A part that exits 0 with no JUnit test count is now red (125) — justified
  8. New linux-part-logs-<g> artifact, uploaded on always() — justified
  9. The cron test imports chat_folders_paths early, fixing the first CI run's failure — justified

[FIRST-PRINCIPLES-REVIEWED] 8fa7065

Description read

[DESCRIPTION-READ] f1ab49a32c06ef513da73a53627b2656f6aa5e917d920270e6cbd9bc87d54efe

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 8fa706577e07bee4d8328845a2f635e20cf18c4e and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8fa7065

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 8fa706577e07bee4d8328845a2f635e20cf18c4e false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 8fa706577e07bee4d8328845a2f635e20cf18c4e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 8fa7065

Verdict parsed from the review's SHA-scoped output markers for commit 8fa706577e07bee4d8328845a2f635e20cf18c4e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 8fa706577e07bee4d8328845a2f635e20cf18c4e false-positive: <reason, 20 to 500 characters>

Each Linux backend shard ran one `pytest -n $WORKERS`, so every xdist
worker collected the whole shard before it ran a test: 123-234 s a
worker on main (run 44620769265), and about 1.5-1.8 GiB a worker, which
is what held the count at 6 on an 8-core runner.

The full-suite arm now runs the shard through
scripts/run_sharded_suite.py. The script lists the files the outer
shard owns (same hash, same pin file), hands the list to every part as
--file-shard-targets, and scripts/ci_file_shards.py splits it again by
path hash, so the union of the parts is the shard. Each part is a
single-process pytest with the same plugins, marker filter, timeout and
coverage selectors, and collects only the files it runs. The reduced
and leaf arms are unchanged.

The script gains what a CI shard needs: --outer-shards/--outer-shard/
--outer-pins; --coverage-file, which gives each part its own
COVERAGE_FILE and combines them only when every part passed;
--part-timeout, which kills and reports a part that does not end; and
--print-logs. A part that exits 0 without a JUnit file counting a test
is red, as is a part that ran tests and left no coverage data.

The process count is sized from memory and capped at the cores (one per
1 GiB after 2 GiB for the host): six such processes peaked at 3.5 GiB
resident on shard 1 where six xdist workers peaked at 9.1 GiB.

Related: #39702
@Zedmor
Zedmor force-pushed the ci/linux-shards-run-sharded-suite branch from 6028f59 to d301b89 Compare November 20, 2026 03:16
A bound counted from each part's own start could end after the job's
timeout-minutes, so a part that hung late was cancelled with the job:
never named, its log and stack dump never printed. The script now takes
--run-timeout, one deadline for the whole run; the workflow sets it to
the job's budget less 10 minutes. At the deadline the script kills what
still runs, prints those logs, starts nothing further and exits
non-zero, so the upload steps still run.

A part's output reaches stdout only when the part ends, so the script
also names the running parts once a minute, prints once the log so far
of a part whose log has not grown for six minutes, and prints the log
so far of every running part when a signal or a failed start stops the
run. A new artifact keeps every part's log and JUnit file.

A part's own bound counts from a clock reading taken before the part is
launched, so it is set before anything the part does. The script's
tests read an injected clock and move it only once the thing the move
is meant to trip is on record; the part-bound test holds the launching
thread until the clock has moved, the schedule that would miss a bound
set after the launch.

The first full run failed fifteen tests in one part of shard 7: a cron
test replaced kiro_crew.sel.sel, add_job imported chat_folders_paths
under it, and the stand-in stayed bound for the rest of the process. A
worker that collected the whole shard had imported that module at
collection. The test file now imports it before any test patches.

Related: #39702
@Zedmor
Zedmor force-pushed the ci/linux-shards-run-sharded-suite branch from d301b89 to 6fc8b58 Compare November 20, 2026 03:45
The timing tests of scripts/run_sharded_suite.py drove real children,
FIFOs, helper threads and an injected clock together, and each review
round found another way for that shape to race or hang. They are
replaced by tests that run in the calling thread against the clock
alone: Parts.wait is split into bound(), a non-blocking _ended(), and
_reap(), with the pause between two looks behind a seam, so a stand-in
part ends, or does not, when the test's clock says so. Watch is driven
by calling look() and stopping() directly.

What still starts a real process: one kill of a real part at a deadline
that has already passed (with and without os.waitid), reaped under a
bound on the failure path; the coverage combine the script itself runs;
and one nested pytest run under a named lost-run ceiling.

Three of the replaced tests also failed on the first runner that ran
them: they asserted the plain log title, and under GITHUB_ACTIONS the
script folds a log into a group. The tests that read a title now clear
that variable.

The script bounds its own coverage combine, the one child it waited for
without a limit.

Related: #39702
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 20, 2026
Akim Akimov added 2 commits November 20, 2026 05:20
Defects in the tests the previous commit added, fixed as classes.

Text mode: two tests compared a log's bytes with text written in text
mode, which is CRLF on Windows; Windows shard 4 failed on both. Every
file a test here writes and then compares byte for byte is written as
bytes.

Real children: the tests of this change now start real processes in
one place, a module-scoped fixture that runs two pytest parts and the
script's coverage combine once, under a named 60 s run ceiling and a
named 60 s combine ceiling, inside the 300 s its consumers allow; a
part or a combine that does not end is killed and named by the script
before pytest-timeout would end the worker. The separate real-kill test
and the second, red nested run are gone; those verdicts are covered
without a process. The module carries one xdist_group, so every real
child in it, the earlier tests' included, starts on one worker.

The test that ran a real combine under a 120 s timeout against the
script's 300 s deadline now checks which parts are combined through a
stand-in, and the combine's stand-in is set on the script's own name
for it, not on subprocess.run.

Related: #39702
One conflict, in the job table of docs/ci/ci-and-reviews.md: main changed
the backend-test-windows row (#39666) next to the backend-test row this
branch changes. Main's rows are kept and this branch's backend-test row
is put back.
@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 20, 2026
@Zedmor

Zedmor commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Five full CI runs with all 8 Linux backend shards green are in. Four are on the current head (8fa7065, run 44682914304 attempts 1-4, Windows also 8 of 8 green each time); the first is on an earlier head (9e67e6f, run 44674727926), where Linux was green and Windows shard 4 was red on two of this PR's own tests, since fixed.

Linux Run tests step, 8 shards, 8 processes and 48 parts per shard:

Run Head Linux shards green Run tests summed Slowest shard Windows shards green
44674727926 9e67e6f 8 of 8 8,455 s 1,179 s 7 of 8
44682914304 attempt 1 8fa7065 8 of 8 8,158 s 1,164 s 8 of 8
44682914304 attempt 2 8fa7065 8 of 8 8,174 s 1,185 s 8 of 8
44682914304 attempt 3 8fa7065 8 of 8 7,798 s 1,167 s 8 of 8
44682914304 attempt 4 8fa7065 8 of 8 7,807 s 1,169 s 8 of 8

Main push runs from the same morning (pytest -n 6 per shard), all 8 Linux shards green:

Main run Run tests summed Slowest shard
44672595354 9,411 s 1,358 s
44675524626 9,526 s 1,453 s
44681606779 8,504 s 1,186 s
44684993735 8,894 s 1,251 s
44690144712 8,656 s 1,203 s
44691798725 8,407 s 1,226 s

Reading:

  • Mean summed time: 8,078 s here against 8,900 s on main, 0.91x. Mean slowest shard: 1,173 s against 1,280 s, 0.92x.
  • The ranges touch: this PR's slowest run (8,455 s, on the earlier head) is above main's fastest (8,407 s). Taking only the four runs on the current head (7,798-8,174 s) against the four latest main runs (8,407-8,894 s) they do not overlap, and the ratio is 0.93x.
  • So the gain on fleet runners is about 7-9%, against 0.71x measured locally before the PR. The slowest shard stays at about 19.5 minutes; the issue's 15-minute target is not met by this change alone.
  • Coverage on the green runs equals main (94.30%, same per-file floor counts). Peak memory stayed under 6.9 GiB of 15.7 GiB at 8 processes.
  • One hidden test order dependence surfaced during these runs and is fixed in this PR (test_cron_auto_pause_limit.py). The suite has not been swept for other instances; a different part count or pin file can surface another.

The PR stays a draft. Open for the maintainer: whether a 7-9% gain is worth the added runner script and the order-dependence risk, and the branch has 5 commits (4 plus one merge of main) because a squash needs a force-push.

AI-assisted, posted by Claude Code

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 20, 2026
@Zedmor

Zedmor commented Nov 20, 2026

Copy link
Copy Markdown
Contributor Author

Close-out: measured, explained, and a recommendation not to ship this

Recommendation: close this PR without merging. On the fleet runner it buys about 9-13 % of Linux Run tests time, not the 29 % measured locally, the best variant I could find is predicted to reach about 15 % with a wide error bar, and every variant keeps a risk class main does not have. A lever on main with a similar predicted gain and no change to the process model is named at the end. I have not closed the PR; that is the maintainer's call.

What CI gave

runs Run tests, summed over 8 Linux shards slowest shard
main, same morning (44681606779, 44675524626, 44672595354) 3 8,504 / 9,526 / 9,411 s 1,186 / 1,453 / 1,358 s
this PR, 8 processes x 48 parts (44661146070, 44669849461, 44674727926, 44682914304) 4 7,514 / 7,682 / 8,455 / 8,158 s 1,115 / 1,146 / 1,179 / 1,164 s

Means 7,952 s against 9,147 s (0.87x); the ranges touch. Slowest shard about 19 minutes against the issue's target of 15. Coverage was equal (94.30 % both, same per-file floor counts), peak memory 4.3-6.9 GiB of 15.7 GiB.

Why CI gets less than the local 0.71x

Measured on one pair of runs of the same morning with the same files in every shard (main 44681606779, this PR 44682914304), from the linux-pytest-progress-* artifacts:

  1. Test seconds grow by 19 % when 8 processes replace 6 workers. The same files took 39,550 s of test phases on main and 46,950 s here (+18.7 %; per shard from +0 % to +48 %). Six to eight processes is 0.75x at best; times 1.187 it is 0.89x. Locally, on 8 pinned cores, there was no growth (3,512-4,070 s old, 3,150-3,594 s new). The development host has one thread per core (lscpu); the fleet runner is "8 vCPU", and on EC2 a vCPU is by default one hardware thread of a core with two (EC2 CPU options). I did not read the runner's topology, so SMT is the likely cause, not a verified one; lscpu in one job would settle it.
  2. Collection fell by 13 %, not by five sixths. Summed over all processes it went from 8,316 s to 7,227 s. A part pays a fixed cost whatever it owns: fitted over the 384 parts of run 44661146070, collection is 12.2 s + 0.21 s per file, plus 5 s outside the session (start, exit, coverage save) and about 4 s inside it. That is about 21 s a part, about 8,000 process-seconds a run, about 1,000 s of the summed step time.
  3. The end of each shard idles. The parts are a hash split; 42-77 s per shard pass with fewer than 8 processes busy. One file, test/test_facade_shared_checks.py, takes 487-723 s alone, and a single-process part cannot split a file.

For scale, main's 51,024 process-seconds in that run are 77.5 % test phases, 16.3 % collection and 6.2 % everything else.

What was tried, and what each would have needed to show

variant evidence result verdict
8 processes, 48 hash parts (this PR) 4 CI runs 0.87x of same-morning main, ranges touch below the bar
6 processes, same count as main local, 32 cores and 8 pinned cores 0.96x and 0.93x no gain: the fixed cost per part eats the single collection
8 parts per shard, planned from recorded seconds with the pin generator's place() replay on a held-out run: 7,065 s against 7,598 s for hash parts (-7 %); local, 8 pinned cores: 617 / 651 s against 676 / 671 s (-6 %) predicted about 0.85x of main on CI at the bar inside the noise; adds a planner and a committed weights file; the giant file caps it
7 processes not run 7 would need the growth in test seconds to fall from 19 % to under 4 % to beat 8 not worth a CI cycle
no coverage (the ceiling for any tracer change) local, 8 pinned cores 600 s against 676 / 671 s (-11 %) coverage is about a tenth of the step here, not the 33-160 % measured on small slices
COVERAGE_CORE=sysmon coverage.py configuration, [run] core "In Python 3.12 and 3.13, it does not support branch coverage"; this repo measures branches on 3.12 not available; on 3.14 it is the default, but one suite reports it slower than the C tracer there (coveragepy#2172), and its author wrote "faster for line coverage, but not yet for branch coverage" (post)
keep xdist, collect less per worker pytest-xdist, how it works workers "perform a full test collection" and the controller checks "that all workers collected the same tests (including order), bailing out otherwise" not possible inside xdist
more shards, same work arithmetic on the table above job-minutes unchanged (plus one set-up per added shard); slowest shard scales with 1/N the only lever here for the 15-minute target, see below

The risk class this design adds

A part imports far fewer modules at collection than a worker that collects a whole shard, so more of the package is first imported in the middle of a test. The first CI run found one such case: test_cron_auto_pause_limit.py replaces kiro_crew.sel.sel, add_job imports chat_folders_paths under it, and 15 later tests in the same process fail at the chat_folders facade's import. It is deterministic for a given split, so every change of part count, pin file or shard count can surface another pair. The planned-parts variant does not remove it. That defect is on main independently of this PR (pytest -n0 test/test_cron_auto_pause_limit.py test/test_power.py failed on main at 49e045e); the six-line fix in this branch is worth landing by itself.

What I recommend instead

  1. Make the per-test fixture floor cheaper (not weaker). In main run 44681606779, of 39,557 s of test phases, setup is 9,183 s and teardown 4,345 s: 34 % of all test time is fixtures, about 46 ms per test over 295,000 tests, while the median test body takes 3.2 ms (median setup 24.7 ms). perf(test): share each collected item's fixture bookkeeping #37814 counted about 130 fixtures resolved per item, 122 of them autouse. Locally 2,000 no-op tests take 24-26 s without coverage and 42-47 s with it. Halving that floor removes about 6,700 process-seconds a run: about 1,100 s of the summed Linux Run tests (about 13 %), the same share on Windows and macOS, with no change to how tests are grouped into processes. Falsifier: a profile of a no-op test that shows no fixture worth more than a fraction of a millisecond.
  2. The slowest 1 % of tests carry 51 % of the test-body seconds (about 13,000 s a run). The progress artifacts name them.
  3. For the slowest-shard target, raise the shard count. At 12 shards the slowest shard's step is about two thirds of today's, about 13 minutes; job-minutes stay about the same. This is the change the issue says waits on a runner-quota decision.
  4. If this design is ever revisited: read the runner's lscpu first, and use the planned 8 parts per shard rather than 48 hash parts.

Sources for the method: per-process collection cost in large suites is a known shape, for example a global conftest.py that makes an empty test take about 13 s (Discord, pytest daemon).

Not verified: the runner's CPU topology; the planned-parts and no-coverage variants on CI (local and replay only); the fixture-floor estimate beyond the split of phase seconds and the no-op timing.

AI-assisted, posted by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant