Skip to content

feat(brainchip_utils): save benchmark results to file and redraw the plots from it - #118

Open
dmclellandbc wants to merge 2 commits into
mainfrom
feat/HUB-40-benchmark-results-file
Open

dmclellandbc wants to merge 2 commits into
mainfrom
feat/HUB-40-benchmark-results-file

Conversation

@dmclellandbc

@dmclellandbc dmclellandbc commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

Refs HUB-40

Hardware benchmarks take minutes to hours, but their figures only need the numbers they measured. Until now those numbers weren't kept, so any change to how the benchmark plots look meant re-running the benchmarks. This PR saves them to a results file next to the figures, so the figures can be redrawn without a device. It's built in one example, akida1/model_zoo/vww, and the other examples follow separately.

What changes

  • New brainchip_utils/benchmark_results.py. save_benchmark_results() writes a JSON file with everything both figures need:

    • full-model timing and power summary for each map mode;
    • each mode's hardware mapping (NPs per layer, pass and sequence boundaries);
    • per-layer timing, including the raw cumulative clocks and times behind the per-layer spread;
    • per-layer sparsity, the figure titles and file names, and provenance (device, akida version, clock).

    plot_benchmark_results() draws the figures from that file, and there's a command line for it:

    python -m brainchip_utils.benchmark_results akida1/model_zoo/vww/docs/ref_benchmark_results.json

    The module doesn't import akida, so redrawing works on a machine without it (tested).

  • Power traces are stored as the plot uses them. Each repeat is resampled onto the plot's shared 500-point time grid, and the raw sensor readings aren't kept. Those readings run to hundreds of thousands of points per map mode, while the resampled traces take about 40 KB per mode.

  • Layers the per-layer benchmark skips are stored as null. Their timings are NaN, which is invalid JSON, so the file stays strict JSON, and they're read back as NaN. This is ready for models with skip connections, whose Add layers can't be timed on their own.

  • plot_utils:

    • The plot helpers accept a saved mapping (from the new mapping_summary()) or a saved power trace (from power_trace()), as well as a live model or raw readings.
    • plot_full_model_results re-maps the model only for results that carry no mapping.
    • Existing callers in scripts and notebooks are unchanged.
    • import akida is now only done where it's needed.
  • akida1/model_zoo/vww/vww_benchmark.py: records each mode's mapping during the benchmark loop, saves the results file (to docs/ref_benchmark_results.json with --save-metrics), then draws the figures from the saved results. So the figures drawn during a run are exactly the ones the file reproduces later.

Verification

  • Unit tests. test/test_benchmark_results.py covers save → load → plot with synthetic results, including power traces. It also checks that figures drawn from saved results match figures drawn from the live data (lines, bars and labels), that skipped (NaN) layers are stored as null and read back as NaN, and that the module imports without akida. The full unit suite passes (39 tests), and check_content.py reports nothing.
  • One hardware run: akidanet_vww_qat.fbz on an AKD1500 (PCIe, I²C power sensing), akida 2.19.3, run without --save-metrics from a scratch directory. It took about 5 minutes. A wrapper also drew the figures the way main does (raw power readings, live re-map for each mode) to compare against:
    • Live run vs redrawn from the file: pixel-identical, for both figures, and also when redrawn on a different machine.
    • main-style vs redrawn from the file: the per-layer figure is pixel-identical. In the full-model figure, 0.3 % of pixels differ, by at most 2.7 % intensity: anti-aliasing from rounding the stored power traces to 0.01 mW.
    • File size: 354 KB, of which 235 KB is the raw per-layer arrays (15 layers × 1000 repeats). It's plain JSON, committed without LFS.

Not in this PR

  • No benchmark was re-run for the repo. No committed figures, metrics or READMEs change, and no results file is committed yet. The VWW results file and figures arrive with the next scheduled benchmark re-run, which is also when the README's maintenance section should mention the redraw command.
  • Other examples follow in separate PRs. Those are the other Akida 1 and Akida 2 benchmark scripts. Akida 2 has no power data, so its files are a subset of this format.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DnyD76hgk85WiSBvfGpN6w

dmclellandbc and others added 2 commits October 21, 2026 20:17
…ts from it

The Akida 1 VWW benchmark now writes the numbers behind its plots to a JSON
results file (full-model timing and power per map mode, each mode's mapping,
per-layer timing and sparsity), and draws the figures from that file, so they
can be redrawn later without hardware:

    python -m brainchip_utils.benchmark_results docs/ref_benchmark_results.json

Power is stored as the plot resamples it (one trace per repeat on a shared
grid), not as raw sensor readings. The plot helpers accept a saved mapping or
power trace as well as a live model or raw readings, and plot_utils no longer
imports akida, so existing callers are unchanged.

Refs HUB-40

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnyD76hgk85WiSBvfGpN6w
…hmark results

NaN timings (a layer the per-layer benchmark can't measure on its own, as for
models with skip connections) are written as JSON null and read back as NaN,
so the file stays strict JSON.

Refs HUB-40

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnyD76hgk85WiSBvfGpN6w
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant