Skip to content

Speed up frontend CI artifact delivery with measured Bun benchmarks - #2029

Draft
David Pine (IEvangelist) wants to merge 3 commits into
mainfrom
ievangelist-bun-build-performance
Draft

David Pine (IEvangelist) wants to merge 3 commits into
mainfrom
ievangelist-bun-build-performance

Conversation

@IEvangelist

@IEvangelist David Pine (IEvangelist) commented Nov 2, 2026 •

Copy link
Copy Markdown
Member

Summary

Controlled performance exploration with one evidence-backed production improvement: move the browsable HTML artifact upload off the E2E critical path, while preserving the original artifact, PR comment, partial failed-build output, and every test gate.

  • Transport the complete site as tar+zstd (frontend-e2e-dist).
  • E2E and a separate standard-runner publisher consume the same archive concurrently.
  • The publisher recreates the unchanged browsable frontend-dist artifact and PR comment. It is required by the frontend gate.
  • The first production candidate packed in 6 seconds and uploaded in 4 seconds, versus the baseline's 119-second HTML upload. The HTML upload still happens, just no longer delays browser-test startup.
  • Keep Vitest on Node, keep the current Bun version and install layout, and do not persist a transpiler cache: measurements do not support changing them.

The separate experiment workflow compares complete Vitest inventories on Node/Bun in alternating order, probes native Bun mocks, measures frozen-lockfile installation layouts, profiles actual Bun builds with/without transpiler caching, and tests browser worker counts. Warmup and CPU-profiled rounds are excluded from warm test timings. Experiments run on PR open/reopen or manual dispatch, not every push.

Baseline: https://github.com/microsoft/aspire.dev/actions/runs/41465710789 . Installs ~2s, build 298s, artifact upload 119s, slowest E2E shard 899s.

Measurements

First experimental run: https://github.com/microsoft/aspire.dev/actions/runs/41504904277 .

Experiment Observation Decision
Node 24 + full Vitest suite Warm runs 61.60s / 61.44s; 1,163 passed, 1 skipped Valid control
Bun 1.4.3 + full Vitest suite Warm runs 50.58s / 50.57s; 1,161 passed, 2 failed, 1 skipped Not eligible: config re-import fails inside starlight-plugin-icons/Zod
Native bun test Locale tests pass; browser-storage tests fail on unsupported global mocks Not a drop-in replacement
Warm installs: hoisted / isolated / global ~0.30-0.33s / 0.18s / 0.04-0.07s; contract tests pass Sub-second CI opportunity; independent cold-cache verification below
Worktree dependency directory 707 MiB hoisted, 720 MiB isolated, 68 MiB global Potentially useful locally; shared store is additional disk use
Full-site tar+zstd 8.52 GB output to 335 MB archive; 8.82s packing, 6s upload; byte-for-byte extraction passed Strong producer-upload candidate; download 4s + extraction 43s, so do not claim every shard downloads faster
Bun profiled builds: cache enabled / disabled 370s / 416s including profiler shutdown; same 68,384 paths; peak RSS ~25 GiB Inconclusive; repeated warm-state comparison below reversed the result

Second run: https://github.com/microsoft/aspire.dev/actions/runs/41506459350 .

  • Latest stable Bun 1.4.4 still fails the same two config re-import tests. Its warm runs were 37.02s / 39.53s versus Node's 42.06s / 43.13s on that runner. Faster incomplete coverage is not eligible.
  • Global store verified 829 actual shared entries, rather than assuming an accepted environment flag proves the setting worked. Independent cold installs were hoisted 1.79s, isolated 1.81s, global 1.68s; subsequent installs were 0.64-0.71s, 0.26-0.31s, and 0.05-0.26s respectively. All layouts passed 13 contract tests. Worth exploring for worktree disk sharing, but not a meaningful CI critical-path improvement.
  • The repeated build comparison reversed: cache enabled 393s, cache disabled 272s. This is strong evidence against interpreting the first pair as a cache win. Do not add a cache action on this evidence.
  • Browser worker comparison verified identical 255-test inventories. Two workers: 14m36s, 255 passed, zero retries. Four workers: 19m37s, 249 passed, six failed, 12 retries. Failures include navigation transitions and existing 60s/120s timeouts. Reject four workers: it was slower and less reliable. Keep production workers, timeouts, retries, assertions, and browser runtime unchanged.
  • Profiling is explicitly separated from warm test timings. Build CPU samples include native/blocking work and must not be interpreted as exact self-CPU time for every function.

Additional Windows compatibility probe: temporarily inlining Zod through Vite makes both Bun config re-import tests pass, but the complete Bun suite then fails the twoslash generator's tsx subprocess (preload not found for its loader file URL; 1,151 passed, 13 skipped, one failed suite). The temporary inlining change was removed. Replacing the test runtime is not yet a portable drop-in change.

Production candidate CI: https://github.com/microsoft/aspire.dev/actions/runs/41506955689 passed: build, lint/unit validation, AppHost, publisher/PR comment, all six E2E shards, merged report, frontend gate, and CI gate. All assertions, viewport/shard coverage, timeouts, retries, and reporters remain intact.

The candidate's complete CI span was 24m28s versus the baseline's 26m09s. This is an observed comparison, not a controlled estimate of total speedup across different runner/cache conditions. The directly measured improvement is producer packing/upload: 10s instead of 119s. Consumer download plus extraction ranged 24-58s, so this does not promise faster transfer on every shard. The legacy HTML upload took 158s on its separate runner and no longer blocked browser startup.

Third-party links and affiliations

None. Reuses existing pinned actions and runner-provided tar/zstd; no new dependencies.

Validation

  • actionlint .github/workflows/frontend-build.yml .github/workflows/frontend-runtime-experiments.yml: passed.
  • node node_modules/eslint/bin/eslint.js scripts/benchmark-test-runtimes.mjs config/build-timing.mjs tests/unit/build-performance.vitest.test.ts --max-warnings 0: passed, from src/frontend.
  • node --check src/frontend/scripts/benchmark-test-runtimes.mjs: passed.
  • node node_modules/vitest/vitest.mjs run --config vitest.config.ts tests/unit/build-performance.vitest.test.ts tests/unit/astro-vite-config.vitest.test.ts: 11 passed, from src/frontend after restoring the unchanged runtime configuration.
  • node scripts/benchmark-test-runtimes.mjs: Node controls passed on CI; Bun 1.4.3 and 1.4.4 candidates rejected on the two config failures. The native Bun probe also records the unsupported global-mock failures. Candidate failures are retained in artifacts rather than silently skipped; the four-worker experiment failed visibly.
  • diff -qr dist extracted: passed in both CI experiment runs.
  • bun run lint, bun run test:unit: passed in production-candidate CI.
  • bun run build:production: passed on CI only.
  • bun run playwright test --project=desktop-chromium --shard=1/2 and bun run playwright test --project=desktop-chromium --shard=2/2: both passed on CI.
  • bun run playwright test --project=tablet-chromium --shard=1/2 and bun run playwright test --project=tablet-chromium --shard=2/2: both passed on CI.
  • bun run playwright test --project=mobile-chromium --shard=1/2 and bun run playwright test --project=mobile-chromium --shard=2/2: both passed on CI.
  • bun run playwright merge-reports --reporter=html ./all-blob-reports: passed on CI.
  • Both experiment runs completed full production builds and lossless artifact comparisons on CI; no local site build was run.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

Frontend HTML artifact ready

The latest frontend build uploaded the frontend-dist artifact for PR #2029. Use the VS Code button below to open this PR with GitHub Artifacts Explorer and browse the built HTML locally.

VS Code: Open PR #2029 artifacts

This comment updates automatically when a new frontend build artifact is uploaded.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Keep the original frontend-dist HTML artifact and PR comment, while transporting the full site once as tar+zstd for E2E and publication. Preserve failed-build output and require publication in the frontend gate.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@IEvangelist David Pine (IEvangelist) changed the title Measure Bun test runtimes, build caches, and lossless CI artifact delivery Speed up frontend CI artifact delivery with measured Bun benchmarks Nov 2, 2026

Copy link
Copy Markdown
Member Author

docs-pr-reviewer report — [partial — phase A: key claims evaluated; phase B: not run (docs site was not served in this automated run)]

Recommendation: needs human

Source used: n/a — repository CI workflow (frontend artifact delivery, Bun benchmarks)
PR status: draft, mergeable, maintainer-authored (IEvangelist), opened 2026-11-02 — not stale.

Phase A — claims

  • ❓ CI performance change (compressed site artifact, separate runner for browsable HTML). No docs claims; maintainer review of workflow correctness and measured benchmark claims required; still a draft.

Phase B — doc-tester

Not run; rendering of the changed pages is not browser-checked.

🤖 Posted by aspire.dev - squad (AI Agent)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant