Repository navigation
Akida 2: hand gesture recognition (Jester) with a streaming TENN - #120
Open
mtorresBrainChip wants to merge 5 commits into
Open
mtorresBrainChip wants to merge 5 commits into
mtorresBrainChip wants to merge 5 commits into
Conversation
…eval-only) Adds an Akida 2 example for BrainChip's published TENN spatiotemporal Jester model: evaluates the float model on whole clips, reproduces the 8-bit quantization and checks it against the published model, converts to Akida and evaluates streaming frame by frame. Adds a Video domain to the Akida 2 landing page and knowledge-base entries on TENN streaming. The FPGA benchmark has not been run yet: the benchmark keys in metrics.json are TBD and the benchmark notebook has no outputs. Refs HUB-44 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e pass Add the Akida 2 FPGA benchmark (per-frame latency in Minimal, AllNps and HwPr), its plots and the benchmark notebook with hardware outputs. Execute the evaluation notebook, describe what the mapping modes show, tidy the prose, and use American spelling throughout the example. Refs HUB-44 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erized_model does Evaluation and sparsity now forward each frame in its own call on a new model per clip, instead of forwarding a clip as one batch. Results are unchanged: per-layer sparsity is identical and the mean is still 56.63%. The README no longer claims that Akida reads the batch axis as time; the benchmark now checks that on hardware and reports it, without depending on it. Drops the knowledge-base and generator-command edits from this branch; they will follow separately once verified on hardware. Refs HUB-44 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eaming Reruns jester_benchmark.py --save-metrics and the benchmark notebook on the Akida 2 FPGA with the code from 7348c32. The hardware streams a clip bit-identically to the software backend, and forwarding a clip as one batch gives the same outputs. Cycle counts moved by less than 0.01%; NP counts, passes and projected latencies are unchanged. Accuracy and sparsity are not affected (they come from jester_eval.py). Drops the model zoo comparison paragraph from the Model Card. Refs HUB-44 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The benchmark table counts the input layer's dedicated input processor as one NP, so its counts are one more than ak_model.summary() and the BrainChip model zoo page report (44 against 43 for Minimal). Refs HUB-44 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
sbruers-brainchip
self-requested a review
November 6, 2026 16:59
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs HUB-44
What this adds
akida2/model_zoo/jester: an eval-only example of BrainChip's TENN spatiotemporal model for hand gesture recognition on Jester, on Akida 2. It has no training stage. It evaluates the published float model, reproduces BrainChip's 8-bit post-training quantization and checks it against the published 8-bit model, converts it to Akida, and benchmarks it on the Akida 2 FPGA.It is the first streaming (temporal) model in the zoo, so the README and notebooks explain how quantization turns the
Conv3Dmodel into a buffered one, and how to evaluate a model that sees one frame at a time:forwardcall;Results
All accuracy figures are on BrainChip's public 541-clip Jester subset, not the full validation split. The README says so wherever they appear.
Per-frame latency on the Akida 2 FPGA (6 nodes, 25 MHz), projected to the 1 GHz AKD2500 target:
NP counts include the input layer's dedicated input processor as one NP, so they are one more than
ak_model.summary()reports. On hardware, a clip streams bit-identically to the software backend. Forwarding a clip as one batch also gives the same outputs as one frame per call.Also in this PR
akida2/README.md: a new Video section in the model zoo table (DOMAINSinakida2/update_readme.py), plus a link from the root README.test/test_models.py: the float quantize-sanity test passesbatch_size(spatiotemporal TENNs need it to calibrate) and builds its random inputs from the quantized model's input shape (one frame, not a clip).How it was run
Evaluation, on the software backend, from
akida2/model_zoo/jester:The quantization reproduction (
jester_eval.sh) gives the same structure and, after conversion, identical integer weights and thresholds as the published model.Hardware benchmark, on the Akida 2 FPGA:
Checks:
update_readme.py: a second run leaves no diff;check_content.py --base origin/main: 0 findings;pytest test/test_hardware_utils.py: 3 passed;pytest test/test_models.py -m "not hardware"on the Jester models: 1 passed.The Jester hardware tests run in
models-hardware.Not run
-dif you have the dataset in BrainChip's TFRecord layout, but that isn't publicly hosted.🤖 Generated with Claude Code