Feature request: Support combining benchmark results

[bdrung]

The following use case happened often enough to trigger me to write this feature request: I often want to benchmark a modification made to the system, but this modification would need some effort to script. So I would like to call hyperfine, then modify the system, then call hyperfine again. Instead of having the result of two individual results, I like to have a nice comparison of those two.

Recent example: I wanted to benchmark the performance impact of the patch submitted in dracut-ng/dracut#1678. So I created a chroot with Dracut installed. Then I ran hyperfine "dracut --debug --force" for the baseline result. Then I applied the submitted patch to /usr/lib/dracut/dracut-logger.sh and ran hyperfine "dracut --debug --force" again. Now I have those results and have to calculate the difference myself:

# hyperfine "dracut --debug --force"
Benchmark 1: dracut --debug --force
  Time (mean ± σ):      8.427 s ±  0.145 s    [User: 12.157 s, System: 4.686 s]
  Range (min … max):    8.208 s …  8.745 s    10 runs
 
# hyperfine "dracut --debug --force"
Benchmark 1: dracut --debug --force
  Time (mean ± σ):      8.306 s ±  0.071 s    [User: 11.887 s, System: 4.624 s]
  Range (min … max):    8.220 s …  8.426 s    10 runs

[al1re3a]

Related native work: #873 proposes --import-json for comparing previously saved runs.

For the separate before/after workflow described here, another option is to export both runs with --export-json, keeping the same command name. The percentage change in mean elapsed time is (after / before - 1) * 100.

I maintain BenchDelta, a small external comparison tool for those two exports. The browser ZIP can be extracted and opened locally; it highlights mean changes and added/removed commands, and exports Markdown or JSON. It does not execute commands or upload benchmark files. It compares means only, not statistical significance, so it is not a substitute for Hyperfine's statistical analysis.