Skip to content

perf(ci): attribute the Run tests step critical path #2478

Backlog

Description

@spboyer

Summary

Recent successful development-branch CI runs spend substantially more time executing tests than building or installing dependencies. This issue is limited to profiling that test step; these observations do not establish that sharding or increasing concurrency will help.

Proposed primary owner: release-engineer.

Observed hosted-run evidence

Observed 2026-11-12 from GitHub job/step start/end timestamps in .github/workflows/squad-ci.yml:

Successful dev run Test job Run tests step Build Root dependency install
43226523596 420 s 301 s 8 s 28 s
43224800067 464 s 328 s 9 s 35 s
43215389047 560 s 394 s 13 s 44 s

Median test execution is 328 seconds; median full test job is 464 seconds. These are the three most recent successful completed dev runs returned at observation time. An intervening cancelled run was excluded, not treated as a successful timing.

Reproduce the timing extraction with gh run view <run-id> --repo github/squad --json jobs,headSha,url, subtracting each job/step's startedAt from completedAt. The runs have different commits/runners and one-second timestamp resolution. This small success-only sample is observational, not controlled A/B evidence or a CI reliability estimate; step medians do not reconstruct workflow elapsed time.

Relevant source at audited revision 47082a81799710c3e457b594123d190c00596ebf: squad-ci.yml:93-195, vitest.config.ts:10-17.

Scope

Parent: github/squad#2481.

Profiling-only: attribute the existing Vitest Run tests step in squad-ci.yml. This issue ends with measured attribution and at most one proposed follow-up, not a broad CI optimization project.

Out of scope: sharding, worker-count tuning, browser/dependency caches, build/release workflows, test removal, permanent scheduling infrastructure and applying an optimization before a specific bottleneck is established.

Measurable acceptance and tests

  • Capture three completed, retry-disabled profiles of the same selected SHA/test inventory on the same hosted-runner class. Record cache state, worker settings, run IDs and failures; do not replace failed attempts invisibly.
  • Publish machine-readable per-file test/collection/transform/setup duration data where available and a top-ten contributor table. Report wall time, overlap and unattributed time explicitly; do not sum overlapping worker durations as elapsed time.
  • Publish selected/executed/passed/failed/skipped counts and coverage inventory alongside the existing workflow baseline. Instrumentation must not change test selection, assertions, coverage or profile/worktree safety guards.
  • Record Run tests elapsed time, total job/workflow elapsed time, Actions minutes and peak resources; any unavailable metric remains unknown rather than zero.
  • Focused collector/report tests cover an absent/truncated profile, nonzero test exit and overlapping durations; failures cannot become success-shaped reports.
  • Close with one evidenced bottleneck and a separately scoped optimization proposal, or a documented no-material-candidate result. No speedup is required or claimed by this attribution issue; implementation needs its own focused acceptance criteria.

Related work and non-goals

Closed github/squad#872 addressed runner consolidation; github/squad#2284 addressed timeout/browser-cache mitigation. This is a focused execution-critical-path investigation, not reopening those changes or prescribing blind parallelism.

Methodology: microsoft/waza#846 and dotnet/aspnetcore#70219.

Activity

  1. changed the title [-]perf(ci): profile and reduce the test-suite critical path without weakening coverage[/-] [+]perf(ci): attribute the Run tests step critical path[/+] on Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions