Skip to content

Add CodSpeed performance benchmarking setup - #185

Closed
codspeed-hq[bot] wants to merge 3 commits into
mainfrom
codspeed-wizard-1784448736678
Closed

Add CodSpeed performance benchmarking setup#185
codspeed-hq[bot] wants to merge 3 commits into
mainfrom
codspeed-wizard-1784448736678

Conversation

@codspeed-hq

@codspeed-hq codspeed-hq Bot commented Jul 19, 2026

Copy link
Copy Markdown

Summary

This PR integrates CodSpeed for continuous performance measurement of the alexandrite compiler. It reuses the project's existing Criterion benchmarks by swapping the criterion crate for the codspeed-criterion-compat compatibility layer, so no benchmark code needed to change.

Changes

  • tests-compatibility/Cargo.toml and compiler-core/documenting/Cargo.toml: replaced the criterion dev-dependency with codspeed-criterion-compat (aliased as criterion). The html_reports feature is preserved. In tests-compatibility, criterion was also moved from [dependencies] to [dev-dependencies], where it is actually used (only in benches/).
  • Cargo.lock: updated to reflect the new dependency.
  • .github/workflows/codspeed.yml: new workflow that builds and runs the benchmarks under CodSpeed in simulation mode on push to main, on pull requests, and on manual dispatch. It authenticates via OpenID Connect (id-token: write).
  • README.md: added the CodSpeed badge.

Benchmarks included

The workflow builds and runs the benchmarks that are fully self-contained and deterministic, which makes them a good fit for CodSpeed's CPU simulation instrument:

  • documenting (document-module-synthetic) — synthetic module, no external inputs.
  • tests-compatibility parsing (lex / layout / parse, single- and multi-core) — runs against the in-repo tests-integration/fixtures corpus.
  • tests-compatibility stabilizing (stabilize-single-core) — same corpus.

The checking_single_core / checking_multi_core benchmarks are intentionally left out of CI: they depend on external PureScript package sets that must be downloaded and prepared beforehand (cargo run -p tests-compatibility -- prepare ...), and they use long fixed measurement windows (60s+). Those characteristics make them unsuitable for the CI benchmarking loop. They continue to work locally through the Criterion harness unchanged.

Notes

  • The tests-compatibility crate depends on git2, which links against OpenSSL, so the workflow installs libssl-dev and pkg-config before building.
  • Runners: this repository is under a personal account, so the workflow uses the standard ubuntu-latest runner. Simulation mode does not require macro runners, since measurements come from deterministic CPU simulation rather than wall-clock timing.

Verification

All included benchmarks were built and run locally with the CodSpeed CLI in simulation mode, and results were produced and uploaded successfully for documenting, parsing, and stabilizing.

Next steps

  • Merge to start tracking performance on every PR.
  • Optionally extend coverage to the checking_* benchmarks by adding a prepared package-set cache step to the workflow if end-to-end checking performance becomes a priority.

@codecov

codecov Bot commented Jul 19, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.37%. Comparing base (48bf6a4) to head (c9b81ec).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #185      +/-   ##
==========================================
- Coverage   83.40%   83.37%   -0.04%     
==========================================
  Files         145      145              
  Lines       27750    27750              
  Branches    27750    27750              
==========================================
- Hits        23146    23137       -9     
- Misses       2477     2481       +4     
- Partials     2127     2132       +5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@codspeed-hq

codspeed-hq Bot commented Jul 19, 2026

Copy link
Copy Markdown
Author

Congrats! CodSpeed is installed 🎉

🆕 8 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks


Open in CodSpeed

@codspeed-hq
codspeed-hq Bot marked this pull request as ready for review July 19, 2026 08:41
@codspeed-hq
codspeed-hq Bot requested a review from purefunctor July 19, 2026 08:42
@purefunctor
purefunctor deleted the codspeed-wizard-1784448736678 branch August 3, 2026 05:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants