Skip to content

ci: run the windows backend shards with four workers again - #36667

Closed
bolichen97 wants to merge 1 commit into
mainfrom
fix/30257-windows-shard-workers
Closed

bolichen97 wants to merge 1 commit into
mainfrom
fix/30257-windows-shard-workers

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Windows backend test shards get back the 4 xdist workers the memory budget took away, so they stop setting every PR's CI wall clock.

backend-test-windows runs its 8 file shards with pytest -n auto, which resolves through xdist_budget.resolve_workers(). #33084 raised the Linux/Windows per-worker reservation (_GIB_PER_WORKER, _GIB_PER_WORKER_AVAILABLE) from 3 to 5 GiB, for good reasons on developer hosts. A side effect is that the 15 GiB / 8-core Windows runner now gets 2 workers instead of 4:

before (2026-11-02, 2026-11-09 04:16Z): xdist worker budget: 4 of 8 workers (12.3 GiB free, 15 GiB installed) / created: 4/4 workers
after  (every sampled shard Nov 9-11):  xdist worker budget: 2 of 8 workers (12.2 GiB free, 15 GiB installed) / created: 2/2 workers

Each Windows shard's test step went from 22.9-25.6 min to about 42 min.

Why it matters

Those Windows shards are now the last job to finish on every CodeBuild-routed PR run. The median PR wall clock went from 36.6 to 51.1 min, so every contributor waits about 15 extra minutes for a verdict on each push. The Linux shards are unaffected because they already pass an explicit -n "$WORKERS".

Not a goal

  • Changing the xdist memory budget (xdist_budget.py, its 5 GiB per-worker constants).
  • Changing SHARD_COUNT, any needs:, or lint-gate.
  • Changing any other -n auto job (new-test-stability and the rest; none of them is on the critical path).

What changed (motivation → approach → change)

A PR's CI verdict should arrive about 12 minutes sooner (51 min to about 39): the Windows backend test shards get back their 4 test workers, so they finish before the Linux path again instead of last. The rest of the 15 minutes is Linux suite growth, which this PR does not touch.

  • Symptom: Windows shards take about 42 min and set the PR wall clock.
  • Cause: -n auto on that runner goes through the memory budget, and at 5 GiB per worker 15 GiB installed / 12.2 GiB free allows 2 workers.
  • Change: in .github/workflows/ci.yml, step "Run tests (Windows shard N/8)" of backend-test-windows, -n auto becomes -n 4. That is the count the same runner got, and ran green with, before fix: guard host memory against parallel test runs #33084. This follows the Linux lane, which already passes an explicit count. An explicit -n still claims its slots from the shared pool through xdist_budget.claim_explicit_workers, which keeps N for a run that is alone on its host (a CI runner always is). A two-line comment above the pytest call says why the count is explicit. The comment block above the step said "-n auto already saturates each runner", which is no longer accurate, so it now says "The shard's workers already saturate each runner".
  • test/test_ci_windows_canary.py::test_shard_selection_and_checks_stay_intact (exact command string) and test/test_ci_pytest_progress.py::TestCIProgress::test_windows_wiring_keeps_acceptance_and_other_jobs_unchanged (flag list) now pin -n 4. The latter also asserts -n auto is absent from the Windows command, as the Linux wiring test already does.

Backwards compatibility

Compatible: a CI-only worker count on one job; no product code, config key or API changes.

Removes nothing: the step still runs every test it ran, only with an explicit worker count.

Tests

  • test_shard_selection_and_checks_stay_intact pins the Windows step's full pytest command line, now with -n 4.
  • test_windows_wiring_keeps_acceptance_and_other_jobs_unchanged pins -n 4 in the flag list and that -n auto is absent.
  • Both fail against main's ci.yml (assert ... -n 4 ... == ... -n auto ...) and pass against this branch's.

Proof: not provable -- docs or config only

Manual verification

Host: Linux (Linux 7.0.0-1014-aws x86_64). The real Windows-runner proof is this PR's own CI run: its Windows shard logs must show created: 4/4 workers.

How it was verified

  1. Reproduced the worker grant with the real budget code, patching only the machine readings to the Windows runner's (8 cores, 15 GiB installed, 12.2 GiB available, no cgroup, fresh KIROCREW_TEST_SLOT_DIR):
_GIB_PER_WORKER=5 _GIB_PER_WORKER_AVAILABLE=5
UserWarning: xdist worker budget: 2 of 8 workers (12.2 GiB free, 15 GiB installed). Each worker needs about 5 GiB ...
resolve_workers() with current 5 GiB constants: 2
resolve_workers() with pre-#33084 3 GiB constants: 4
claim_explicit_workers(4), alone on host: 4

The warning line matches the CI shard logs exactly.

  1. Ran the fixed step's pytest command line (-q -n 4 --timeout=180 --no-cov --max-worker-restart=0 -m "not ipv6_required and not real_venv" --file-shards 8 --file-shard N) on this host over test/test_ci_windows_canary.py test/test_ci_pytest_progress.py test/test_ci_file_shards.py test/test_xdist_host_budget.py, for every N:
shard 1 exit=5 :: created: 4/4 workers            (no file of the four lands in shard 1)
shard 3 exit=0 :: created: 4/4 workers  71 passed
shard 6 exit=0 :: created: 4/4 workers  693 passed
shard 7 exit=0 :: created: 4/4 workers  26 passed

Shards 2, 4, 5 and 8 also hold none of the four files (exit 5), and each one created 4/4 workers.

  1. claim_explicit_workers(4) under the Windows-runner readings, after the fix: 4.
  2. python -m pytest -n0 test/test_ci_windows_canary.py test/test_ci_pytest_progress.py test/test_xdist_host_budget.py test/test_workflow_matrix_max_parallel.py -q: 788 passed. All 12 test files that read backend-test-windows from ci.yml, plus test_workflow_matrix_max_parallel.py: 1016 passed.
  3. Gates: black/subprocess-encoding, black, isort, flake8, mypy src/kiro_crew, comment-history, brand and file-line-cap all pass; scripts/local-gate.py: backend related 3659 passed, 2 skipped; frontend related 14760 passed (519 files); "all selected gates passed".
  4. On a second host with the same commit applied: python -m pytest -n0 test/test_flake_pattern_ratchet.py (68 passed), semgrep scan --config semgrep/ --baseline-commit <merge-base> --error (0 findings), npm --prefix website run bench:render -- --fail-on dispatches,selectorCalls (exit 0, its CI filter includes ci.yml).

Related Issues

Part of #30257

The issue stays open for one more measured CI window.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 3cd50f4b8ad134eb68faf5d53846f26183521355 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 3cd50f4b8ad134eb68faf5d53846f26183521355 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this changes the worker count of one CI job, a revert undoes it cleanly, and 4 workers is the count the same runner used, and passed with, before #33084.

Notes

[DESIGN-REVIEWED] 3cd50f4

Description read

[DESCRIPTION-READ] 6b224ddd88914467be315df8849b6e3bc69f71da15202b34dd5b76bb1c78424b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 3cd50f4b8ad134eb68faf5d53846f26183521355 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: this is a CI-only worker count. The 2-worker grant is shown in CI logs and reproduced from the budget code, and the Linux lane already passes an explicit -n.

What this change ships

Inventory (3 items) — 3 justified

Intent: get PR CI verdicts back about 12 minutes sooner by restoring 4 test workers on each Windows backend shard. FIX.

  1. Windows backend shards run with 4 xdist workers instead of the 2 the memory budget grants — justified
  2. The comment above the step drops the stale "-n auto already saturates" wording, and a two-line comment explains why the count is explicit — justified
  3. Two CI wiring tests now pin -n 4, and one also asserts -n auto is absent — justified

[FIRST-PRINCIPLES-REVIEWED] 3cd50f4

Description read

[DESCRIPTION-READ] b71dad788bf33dde103d38c9eed32c7af476a666dbb87170538b398db4cfa956

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 3cd50f4b8ad134eb68faf5d53846f26183521355 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3cd50f4

Verdict parsed from the review's SHA-scoped output markers for commit 3cd50f4b8ad134eb68faf5d53846f26183521355.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3cd50f4b8ad134eb68faf5d53846f26183521355 false-positive: <reason, 20 to 500 characters>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 2c9d93d0a1df4c3271395b1ca943469c0cf8af91 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2c9d93d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2c9d93d0a1df4c3271395b1ca943469c0cf8af91 false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 13, 2026 14:50
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
The Windows backend shards ran `pytest -n auto`, which goes through the
xdist memory budget. At its 5 GiB per-worker reservation (#33084) the
15 GiB / 8-core Windows runner is granted 2 workers instead of 4, which
roughly doubled each shard's run time and made those shards the last
job of every PR's CI run.

Pass an explicit `-n 4`, the count that runner ran green with, as the
Linux lane already passes an explicit count. An explicit `-n` still
claims its slots from the shared pool, and a run alone on its host keeps
all N. The two tests that pin the Windows command now pin `-n 4`, and
one also asserts `-n auto` is absent.

Part of #30257

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@bolichen97
bolichen97 force-pushed the fix/30257-windows-shard-workers branch from 2c9d93d to 3cd50f4 Compare November 13, 2026 20:42
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in a landed fix for test/test_mate_conductor_parity.py::test_every_shared_rule_has_an_ultra_counterpart_or_a_named_gap, so the rebase pulls it in and the Linux shard 3, Windows shard 3 and Coverage Gate reds on 2c9d93d0a1df should go green.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 13, 2026
@buluoray

Copy link
Copy Markdown
Contributor

Review passed on this head. Please rebase onto the latest main so CI can run fresh; this head is 439 commits behind main and its only CI run is from 11-13. A fresh Windows run will also confirm that four workers still hold with the test files added since then. The approval will follow once the rebased head is green.

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: Approve

Why: the decider estimates a 7% chance that this change introduces a defect, within the 12% limit for approving on its own, and every check passes.

CI-only change: backend-test-windows shards pass explicit -n 4 instead of -n auto, restoring the pre-#33084 worker count the 5 GiB/worker budget had cut to 2. Verified claim_explicit_workers keeps N=4 for a lone run (capacity = max(static, min(N, cores))) and stays within cores on 4-core fallbacks. No other test or doc reader pins the Windows -n auto; both pinning tests are updated and would fail on main. All Windows and Linux shards are green on this head; the CodeQL and GPT-lane failures are infrastructure problems unrelated to the change.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 3cd50f4b8a · 11 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] @buluoray the rebase is ready (the same +7/-4 change, no conflict), but I am holding the push until #37964 lands. Main is red right now on test/test_denied_commands_security_ssh_self.py::TestSshFloorNestingDepth::test_the_gate_refuses_a_nest_past_the_bound_without_an_exception (Linux shard 7 and Windows shard 2 of main's run at d2f22f259e). This PR's CI runs on its merge with main, so a push now would come back red for that reason alone. Once #37964 merges I rebase onto it and push; the Windows shard logs on that head will show whether four workers still hold.

@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

This PR and #38921 make the same change: both replace -n auto in the backend-test-windows step "Run tests (Windows shard N/8)" of .github/workflows/ci.yml with an explicit worker count, so the Windows shards get 4 xdist workers again instead of the 2 the 5 GiB budget grants. Both also rewrite the same "-n auto already saturates each runner" comment and edit test/test_ci_pytest_progress.py and test/test_ci_windows_canary.py.

Please keep one of #36667 and #38921 and close the other, carrying over anything only this one has.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 18, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Closing this as superseded by #38921, which merged into main as 577c105 and makes the same change in a fuller form: the Windows backend shard step sizes its own xdist workers (one per 3 GiB after 2 GiB for the host, capped at the core count, so 4 on the 15 GiB fleet runner and 4 on a 4-core hosted runner) instead of -n auto, logs the count and the peak memory, and updates the same two tests that pin the command.

Rebasing this branch onto main conflicts in all three of its files (.github/workflows/ci.yml, test/test_ci_pytest_progress.py, test/test_ci_windows_canary.py), each against #38921's version of the same edit. Resolved to main's side, the merged tree is byte-identical to main's tree (6d10a3f976b3), so this PR has nothing left to carry over and no rebased head to push.

@bolichen97 bolichen97 closed this Nov 18, 2026
auto-merge was automatically disabled November 18, 2026 07:40

Pull request was closed

@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 18, 2026
@bolichen97
bolichen97 deleted the fix/30257-windows-shard-workers branch November 20, 2026 09:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants