Skip to content

ci(macos): balance the nightly macOS shards by measured test time - #37422

Open
CrysisDeu wants to merge 2 commits into
mainfrom
crew/build-watcher/macos-shard-timing
Open

CrysisDeu wants to merge 2 commits into
mainfrom
crew/build-watcher/macos-shard-timing

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Let the nightly macOS backend shards be balanced by measured macOS test time, so the slowest shard drops back under 32 minutes.

The Nightly macOS backend shards (Platform Tests / Backend Tests (macOS) 1..8) have crept up from a 31-minute slowest shard (11/7) to 39 minutes (11/14, run 43564614563), and shard 5 hit the old 40-minute cap on 11/12. pytest time per shard, from the job logs:

shard 11/7 (42345445555) 11/14 (43564614563)
1 1062 s 1342 s
2 1462 s 2245 s
3 977 s 1230 s
4 1076 s 1251 s
5 1793 s 2307 s
6 1101 s 1659 s
7 1183 s 1515 s
8 1252 s 1279 s
mean 1238 s 1603 s (26.7 min)
max / mean 1.45 1.44

Two causes:

  1. The suite grew. Total macOS pytest time went 9906 s -> 12827 s (+29%) in a week; test count +9% (~3,000 tests per shard-set). On Linux the same week added 687 new test files (+2387 s) and +1826 s on existing files (CI runs 42535136833 vs 43649439056). No single test regressed; the growth is spread across many files.
  2. macOS shards are owned by path hash, not by time. Shards 2 and 5 sit ~40% above the mean on both dates, while shard 3 is 23-38% below it. The Linux backend lane fixed this with scripts/ci_file_shard_pins.txt (ci: balance Linux backend test shards with a measured pin file #30278), but that file is built from Linux timings, and those do not predict macOS cost: the same hash split is within 1.11x on Linux. The heavy macOS files are git/subprocess-heavy ones that are cheap on Linux, e.g. test/test_crewmate_worktree.py (19 s on Linux, its top 3 tests alone 208 s on macOS, shard 5) and test/test_remote_command_lifetime.py (2.6 s on Linux, 2 x 60 s on macOS, shard 5).

A balanced split would put every shard near the 26.7-minute mean, under the 32-minute target, without changing the shard count, max-parallel: 4 or the timeout.

Why it matters

The nightly gates publication on these shards. One shard over the cap cancels the run and skips every publish job (11/12). The lane is two waves of 4, so the slowest shard of each wave sets the whole lane's time.

Not a goal

  • Changing SHARD_COUNT, max-parallel (4) or timeout-minutes.
  • Fixing any individual slow macOS test.
  • Generating the macOS pin file itself: that needs the per-file macOS data this PR starts recording. It follows in this PR's second commit once a macOS run of this branch has uploaded the records, or in a follow-up.

What changed (motivation → approach → change)

Symptom: the slowest macOS shard is ~40% over the mean -> cause: path-hash ownership plus macOS-only heavy files that Linux timings cannot see -> change: record per-file macOS seconds the same way backend-test already does, and pin the heaviest files from those measurements.

  • .github/workflows/platform-tests.yml: the macOS shard pytest command loads -p scripts.ci_pytest_progress --ci-progress-dir="$RUNNER_TEMP/pytest-progress" (the same plugin and flag ci.yml backend-test uses), passes --file-shard-pins scripts/ci_file_shard_pins_macos.txt, and a new always() step uploads macos-pytest-progress-<N> (7-day retention) after the existing shard-log upload.
  • scripts/ci_file_shard_pins.py: new --platform {linux,macos} (default linux) choosing which <platform>-pytest-progress-* artifacts to read; the default --out follows it (ci_file_shard_pins.txt / ci_file_shard_pins_macos.txt), so a macOS refresh cannot overwrite the Linux pins. Linux behaviour is unchanged.
  • docs/system-specs/common/testing-conventions.md (Shard balance): macOS now uses its own measured pin file; Windows, release, reduced and leaf keep pure hash ownership.
  • scripts/ci_file_shard_pins_macos.txt: generated, 800 pins, from this PR's two macOS runs (43657254686 hash-only, 43664027179 with a one-run pin file): python scripts/ci_file_shard_pins.py <run-a> <run-b> --platform macos. Replayed on each run's data the pinned worker-seconds per shard are 2763-2998 and 2671-2913, against 2413-3622 for the hash split. Shard count, max-parallel: 4 and timeout-minutes: 60 are unchanged.

Backwards compatibility

Compatible: the plugin only writes timing JSONL and never changes test selection or outcomes; pins change only WHICH macOS shard runs a file, every file still has exactly one owner; the generator's default stays linux.

Removes nothing: the diff only adds a pytest plugin flag, a pin file, an artifact upload step and a CLI option.

Tests

  • test/test_macos_platform_tests_gate.py::test_the_shards_record_per_file_seconds_for_the_pin_generator locks in the plugin flags on the macOS shard command and the macos-pytest-progress-${{ matrix.group }} upload with if: always().
    It also asserts the macOS pin file is wired in, made for SHARD_COUNT, and pins files to every shard.
  • test/test_ci_file_shards.py::test_pin_generator_reads_macos_artifacts_into_the_macos_pin_file locks in that --platform macos reads macos-pytest-progress-*, linux mode refuses them, and the macOS default output is not the Linux pin file.
  • Existing test_the_shard_log_is_uploaded_even_when_the_shard_is_cancelled still finds the shard-log upload first.

Proof: not provable -- test-only change

Manual verification

Labelled ci:macos so macos-on-demand.yml runs the full platform-tests.yml suite on this head; run 43657254686 uploaded all 8 macos-pytest-progress-* artifacts (the pin file's input). Its red shards (1, 2, 4, 8) are main regressions also red on other PRs' macOS runs (test_readonly_source_skill_install.py, test_autonudge_row_scrub.py, test_one_conversation_one_session.py, test_history.py, test_telegram_sessions_resume.py), not this diff. pytest was not run locally.

Related Issues

N/A — found by Build Watcher on the 11/14 nightly (run 43564614563).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners November 15, 2026 00:15
@CrysisDeu CrysisDeu added the ci:macos Run the on-demand macOS suite against this PR head label Nov 15, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 15, 2026
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision a80580a94b5c6c881a0f228e4a37cf2bbaf2927d touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of a80580a94b5c6c881a0f228e4a37cf2bbaf2927d — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change reuses the Linux pin mechanism, writes macOS timings to a separate pin file, and pins change only which shard runs a file, so one revert walks it back.

Notes

  • The 800 pins come from runs where shards 1, 2, 4 and 8 failed. A failed run can stop early, so its timings may undercount some files. Regenerate the pins from green nightly runs once they exist.

[DESIGN-REVIEWED] a80580a

Description read

[DESCRIPTION-READ] 85c58f180092583ee04e60908ed05463f8ba66cdfd8200501647bb29a63c61e3

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of a80580a94b5c6c881a0f228e4a37cf2bbaf2927d — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Every item reuses the existing Linux pin mechanism (--file-shard-pins, ci_pytest_progress, ci_file_shard_pins.py) and is aimed at the measured cause: path-hash ownership on macOS, where Linux timings say nothing about cost.

What this change ships

Inventory (6 items) — 6 justified

Intent: keep the nightly macOS shards under the timeout so publication isn't cancelled; this is a FIX (CI balance).

  1. Nightly macOS shards now record per-file test seconds through the existing progress plugin — justified
  2. Nightly macOS shards now take their file owners from a measured macOS pin file instead of pure path hash — justified
  3. Each macOS shard uploads a macos-pytest-progress-<N> artifact, kept 7 days, even when it fails — justified
  4. The pin generator has a --platform {linux,macos} option; it defaults to linux, and the default output path follows the platform — justified
  5. Generated scripts/ci_file_shard_pins_macos.txt with 800 pins from two macOS runs — justified
  6. The testing-conventions Shard balance section says macOS now has its own pin file — justified

[FIRST-PRINCIPLES-REVIEWED] a80580a

Description read

[DESCRIPTION-READ] 4cca4563cfdb2b53f95460f38621d7e1f919822cf8b25b362501e01e0b84d875

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed a80580a94b5c6c881a0f228e4a37cf2bbaf2927d — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a80580a

Verdict parsed from the review's SHA-scoped output markers for commit a80580a94b5c6c881a0f228e4a37cf2bbaf2927d.

False positive or not applicable? A repository writer can comment:
/ai-review override fable a80580a94b5c6c881a0f228e4a37cf2bbaf2927d false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of a80580a94b5c6c881a0f228e4a37cf2bbaf2927d and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a80580a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt a80580a94b5c6c881a0f228e4a37cf2bbaf2927d false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 902 lines, more than the 400 it may approve on its own.

The PR records per-file macOS test timings with the existing ci_pytest_progress plugin, uploads them as macos-pytest-progress-N, adds --platform to the pin generator (defaults stay linux, and the macOS default output can't overwrite the Linux pin file), and wires a generated 8-shard macOS pin file into the macOS shard command. The pin loader strictly validates shard count, owner range and duplicates, and the new gate test checks the committed file against SHARD_COUNT=8 with every shard getting pins. The timing plugin is payload-only and can't change test selection or outcomes. Pins only move which shard owns a file, so every file still has exactly one owner. Release, Windows and the reduced/leaf scopes are untouched, and the existing tests that limit the progress plugin and pin flags to certain jobs only read ci.yml, so nothing conflicts. Tests cover both new behaviours. Remaining concerns (thin timing data behind the pins, stale comment in test-durations.yml, path filter, description drift) are non-blocking.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit a80580a94b · 902 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@CrysisDeu
CrysisDeu force-pushed the crew/build-watcher/macos-shard-timing branch 2 times, most recently from ff7d960 to 2ad1ad8 Compare November 15, 2026 01:38
@CrysisDeu CrysisDeu changed the title ci(macos): record per-file test seconds in the nightly macOS shards ci(macos): balance the nightly macOS shards by measured test time Nov 15, 2026
@CrysisDeu
CrysisDeu force-pushed the crew/build-watcher/macos-shard-timing branch from 2ad1ad8 to a80580a Compare November 15, 2026 02:40
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 15, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@CrysisDeu 改动本身没问题:权限、secrets、触发条件都没动,新测试本地全过。合并前请做一件事:

  1. Rebase 到最新的 main(git merge-tree 显示没有冲突)。这次 macOS 的 2 片红都已经在 main 上修好了:

    • 分片 4 由 #37476 修(只读源目录在 macOS 上 rename 失败,和本 PR 无关)。
    • 分片 8 由 #37478 修。这是新分片表把 test/test_harness_parity.py 挪到第 8 片后暴露出来的泄漏:同一个 worker 里,前面一个 dashboard 测试在 kiro_crew logger 上留了 handler。

    rebase 后 ci:macos 会再跑一次全套,请确认 8 片全绿。这是新分片表第一次在合并后的树上跑,邻居又变了,可能还有别的泄漏。

  2. 可选:.github/workflows/test-durations.yml 头部注释(第 4-8 行)还写着只有 Linux 用 pin 表,顺手改一下。

另外:Linux Backend Tests (3.12, 7) 是 test_denied_commands_security_cost.py:595 的墙钟时间 flake,Coverage Gate 跟着它红,两者都已重跑转绿。

macOS 全绿后我就批准。

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 15, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

No blocking issues were found on this head. Approval waits on the open maintainer request: rebase onto current main and show a ci:macos run with all 8 shards green. A maintainer will approve after that run.

What the review checked on a80580a:

Done well: the PR reuses the existing Linux pin machinery and adds only a --platform switch. Each platform gets its own default output file, so a macOS refresh cannot overwrite the Linux pins.

Optional:

  • Update the header of .github/workflows/test-durations.yml (lines 4 to 8). It still says only the Linux full suite uses a pin file.
  • The test comment says a default-path macOS refresh never overwrites the Linux pin file, but every main() call passes --out. Add one call without --out, with DEFAULT_OUT patched to temp paths, or trim the comment.
  • After the rebase, link the green run and each shard's time under Manual verification, and replace the "test-only change" proof line. Regenerating the pins from green nightly runs can be a follow-up.

@bolichen97

Copy link
Copy Markdown
Contributor

@CrysisDeu 提醒一下:这个 PR 还需要 rebase 到最新的 main(没有冲突)。rebase 后 ci:macos 会自动跑全套,macOS 8 片全绿我就批准合并。现在 main 上已经连续 3 夜定时 Nightly 一次通过,这个分片平衡合进去后能再多留一些时长余量。

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:macos Run the on-demand macOS suite against this PR head merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants