Add CodSpeed performance benchmarking setup - #185
Closed
codspeed-hq[bot] wants to merge 3 commits into
Closed
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #185 +/- ##
==========================================
- Coverage 83.40% 83.37% -0.04%
==========================================
Files 145 145
Lines 27750 27750
Branches 27750 27750
==========================================
- Hits 23146 23137 -9
- Misses 2477 2481 +4
- Partials 2127 2132 +5 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Author
Congrats! CodSpeed is installed 🎉
You will start to see performance impacts in the reports once the benchmarks are run from your default branch.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR integrates CodSpeed for continuous performance measurement of the alexandrite compiler. It reuses the project's existing Criterion benchmarks by swapping the
criterioncrate for thecodspeed-criterion-compatcompatibility layer, so no benchmark code needed to change.Changes
tests-compatibility/Cargo.tomlandcompiler-core/documenting/Cargo.toml: replaced thecriteriondev-dependency withcodspeed-criterion-compat(aliased ascriterion). Thehtml_reportsfeature is preserved. Intests-compatibility,criterionwas also moved from[dependencies]to[dev-dependencies], where it is actually used (only inbenches/).Cargo.lock: updated to reflect the new dependency..github/workflows/codspeed.yml: new workflow that builds and runs the benchmarks under CodSpeed insimulationmode onpushtomain, on pull requests, and on manual dispatch. It authenticates via OpenID Connect (id-token: write).README.md: added the CodSpeed badge.Benchmarks included
The workflow builds and runs the benchmarks that are fully self-contained and deterministic, which makes them a good fit for CodSpeed's CPU simulation instrument:
documenting(document-module-synthetic) — synthetic module, no external inputs.tests-compatibilityparsing(lex / layout / parse, single- and multi-core) — runs against the in-repotests-integration/fixturescorpus.tests-compatibilitystabilizing(stabilize-single-core) — same corpus.The
checking_single_core/checking_multi_corebenchmarks are intentionally left out of CI: they depend on external PureScript package sets that must be downloaded and prepared beforehand (cargo run -p tests-compatibility -- prepare ...), and they use long fixed measurement windows (60s+). Those characteristics make them unsuitable for the CI benchmarking loop. They continue to work locally through the Criterion harness unchanged.Notes
tests-compatibilitycrate depends ongit2, which links against OpenSSL, so the workflow installslibssl-devandpkg-configbefore building.ubuntu-latestrunner. Simulation mode does not require macro runners, since measurements come from deterministic CPU simulation rather than wall-clock timing.Verification
All included benchmarks were built and run locally with the CodSpeed CLI in simulation mode, and results were produced and uploaded successfully for
documenting,parsing, andstabilizing.Next steps
checking_*benchmarks by adding a prepared package-set cache step to the workflow if end-to-end checking performance becomes a priority.