Repository navigation
feat(brainchip_utils): save benchmark results to file and redraw the plots from it - #118
Open
dmclellandbc wants to merge 2 commits into
Open
dmclellandbc wants to merge 2 commits into
dmclellandbc wants to merge 2 commits into
Conversation
…ts from it
The Akida 1 VWW benchmark now writes the numbers behind its plots to a JSON
results file (full-model timing and power per map mode, each mode's mapping,
per-layer timing and sparsity), and draws the figures from that file, so they
can be redrawn later without hardware:
python -m brainchip_utils.benchmark_results docs/ref_benchmark_results.json
Power is stored as the plot resamples it (one trace per repeat on a shared
grid), not as raw sensor readings. The plot helpers accept a saved mapping or
power trace as well as a live model or raw readings, and plot_utils no longer
imports akida, so existing callers are unchanged.
Refs HUB-40
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnyD76hgk85WiSBvfGpN6w
…hmark results NaN timings (a layer the per-layer benchmark can't measure on its own, as for models with skip connections) are written as JSON null and read back as NaN, so the file stays strict JSON. Refs HUB-40 Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnyD76hgk85WiSBvfGpN6w
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs HUB-40
Hardware benchmarks take minutes to hours, but their figures only need the numbers they measured. Until now those numbers weren't kept, so any change to how the benchmark plots look meant re-running the benchmarks. This PR saves them to a results file next to the figures, so the figures can be redrawn without a device. It's built in one example,
akida1/model_zoo/vww, and the other examples follow separately.What changes
New
brainchip_utils/benchmark_results.py.save_benchmark_results()writes a JSON file with everything both figures need:plot_benchmark_results()draws the figures from that file, and there's a command line for it:The module doesn't import akida, so redrawing works on a machine without it (tested).
Power traces are stored as the plot uses them. Each repeat is resampled onto the plot's shared 500-point time grid, and the raw sensor readings aren't kept. Those readings run to hundreds of thousands of points per map mode, while the resampled traces take about 40 KB per mode.
Layers the per-layer benchmark skips are stored as
null. Their timings are NaN, which is invalid JSON, so the file stays strict JSON, and they're read back as NaN. This is ready for models with skip connections, whose Add layers can't be timed on their own.plot_utils:mapping_summary()) or a saved power trace (frompower_trace()), as well as a live model or raw readings.plot_full_model_resultsre-maps the model only for results that carry no mapping.import akidais now only done where it's needed.akida1/model_zoo/vww/vww_benchmark.py: records each mode's mapping during the benchmark loop, saves the results file (todocs/ref_benchmark_results.jsonwith--save-metrics), then draws the figures from the saved results. So the figures drawn during a run are exactly the ones the file reproduces later.Verification
test/test_benchmark_results.pycovers save → load → plot with synthetic results, including power traces. It also checks that figures drawn from saved results match figures drawn from the live data (lines, bars and labels), that skipped (NaN) layers are stored asnulland read back as NaN, and that the module imports without akida. The full unit suite passes (39 tests), andcheck_content.pyreports nothing.akidanet_vww_qat.fbzon an AKD1500 (PCIe, I²C power sensing), akida 2.19.3, run without--save-metricsfrom a scratch directory. It took about 5 minutes. A wrapper also drew the figures the waymaindoes (raw power readings, live re-map for each mode) to compare against:main-style vs redrawn from the file: the per-layer figure is pixel-identical. In the full-model figure, 0.3 % of pixels differ, by at most 2.7 % intensity: anti-aliasing from rounding the stored power traces to 0.01 mW.Not in this PR
🤖 Generated with Claude Code
https://claude.ai/code/session_01DnyD76hgk85WiSBvfGpN6w