Repository navigation
Description
Summary
Recent successful development-branch CI runs spend substantially more time executing tests than building or installing dependencies. This issue is limited to profiling that test step; these observations do not establish that sharding or increasing concurrency will help.
Proposed primary owner: release-engineer.
Observed hosted-run evidence
Observed 2026-11-12 from GitHub job/step start/end timestamps in .github/workflows/squad-ci.yml:
| Successful dev run | Test job | Run tests step | Build | Root dependency install |
|---|---|---|---|---|
| 43226523596 | 420 s | 301 s | 8 s | 28 s |
| 43224800067 | 464 s | 328 s | 9 s | 35 s |
| 43215389047 | 560 s | 394 s | 13 s | 44 s |
Median test execution is 328 seconds; median full test job is 464 seconds. These are the three most recent successful completed dev runs returned at observation time. An intervening cancelled run was excluded, not treated as a successful timing.
Reproduce the timing extraction with gh run view <run-id> --repo github/squad --json jobs,headSha,url, subtracting each job/step's startedAt from completedAt. The runs have different commits/runners and one-second timestamp resolution. This small success-only sample is observational, not controlled A/B evidence or a CI reliability estimate; step medians do not reconstruct workflow elapsed time.
Relevant source at audited revision 47082a81799710c3e457b594123d190c00596ebf: squad-ci.yml:93-195, vitest.config.ts:10-17.
Scope
Parent: github/squad#2481.
Profiling-only: attribute the existing Vitest Run tests step in squad-ci.yml. This issue ends with measured attribution and at most one proposed follow-up, not a broad CI optimization project.
Out of scope: sharding, worker-count tuning, browser/dependency caches, build/release workflows, test removal, permanent scheduling infrastructure and applying an optimization before a specific bottleneck is established.
Measurable acceptance and tests
- Capture three completed, retry-disabled profiles of the same selected SHA/test inventory on the same hosted-runner class. Record cache state, worker settings, run IDs and failures; do not replace failed attempts invisibly.
- Publish machine-readable per-file test/collection/transform/setup duration data where available and a top-ten contributor table. Report wall time, overlap and unattributed time explicitly; do not sum overlapping worker durations as elapsed time.
- Publish selected/executed/passed/failed/skipped counts and coverage inventory alongside the existing workflow baseline. Instrumentation must not change test selection, assertions, coverage or profile/worktree safety guards.
- Record
Run testselapsed time, total job/workflow elapsed time, Actions minutes and peak resources; any unavailable metric remains unknown rather than zero. - Focused collector/report tests cover an absent/truncated profile, nonzero test exit and overlapping durations; failures cannot become success-shaped reports.
- Close with one evidenced bottleneck and a separately scoped optimization proposal, or a documented no-material-candidate result. No speedup is required or claimed by this attribution issue; implementation needs its own focused acceptance criteria.
Related work and non-goals
Closed github/squad#872 addressed runner consolidation; github/squad#2284 addressed timeout/browser-cache mitigation. This is a focused execution-critical-path investigation, not reopening those changes or prescribing blind parallelism.
Methodology: microsoft/waza#846 and dotnet/aspnetcore#70219.
Activity
[-]perf(ci): profile and reduce the test-suite critical path without weakening coverage[/-][+]perf(ci): attribute the Run tests step critical path[/+]