bench: seed the instruction-count baselines from the first main run - #26
Merged
Merged
Conversation
Copies the merged baselines artifact of the first bench.yml push-to-main run after #21 (run 35281280300 at df482d1) into bench/baselines.json: five keys (m4-softfp, m4f, m33, m55, m55-ooura), two scenarios each (rfft_f32_512, rfft_f32_2048), measured on the vendored Ooura C with arm-none-eabi-gcc 13.2.1 (15:13.2.rel1-2) and QEMU 8.2.2 on ubuntu-24.04. bench/README.md records one row per (key, scenario) with the run URL, the main SHA and the toolchain versions, as the seeding rule requires. From this commit every push and pull request compares against these numbers at +/-3%; the bench.yml run on this PR is the first compare-mode run and must report 0% delta (identical tree). Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ZPTzNxo5Fe4EtpXXKf7Sy
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Copies the merged baselines artifact of the first
bench.ymlpush-to-mainrun after #21 (run 35281280300 atdf482d1) intobench/baselines.json, and records one row per key and scenario inbench/README.mdwith the run URL, themainSHA and the toolchain versions, as the seeding rule in Part 11 of the plan requires. Nothing else changes.Why
From this commit the ratchet is live: every push and pull request compares its instruction counts against these numbers at ±3% per key. The numbers were measured on CI, on the vendored Ooura C (
TAP_DSP_BENCH_ENGINE=reference_c), and are the baseline Stage 2b's port-versus-C comparison is gated on.Two things these numbers say, recorded for the design note rather than changed here: soft-float costs 19× on the M4 (the fixed-point profile's reason to exist), and the Helium backend is 1.8× fewer instructions than Ooura on the M55 at these geometries, not the ~3× the header quotes from an earlier measurement.
Verification
arm-none-eabi-gcc (15:13.2.rel1-2) 13.2.1 20231009,QEMU emulator version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.18), cmake 3.31.6, onubuntu-24.04. The pinned plugin header reportsQEMU_PLUGIN_VERSION 1.bench.ymlrun on this PR is the first compare-mode run: every key is seeded, so it must report 0% delta against an identical tree. That run is the proof this PR needs; nothing can be run locally (no Arm toolchain or QEMU in the container)..textceilings are not set here (text_ceilingstays 0 = report only), per README: they are a wave-2 item. The seeding run measured the M55 float size probe at 107,681 bytes (MinSizeRel,size -A).Notes for the reviewer
maingreen on CI and style for each).🤖 Generated with Claude Code
https://claude.ai/code/session_019ZPTzNxo5Fe4EtpXXKf7Sy
Generated by Claude Code