Skip to content

Akida 2: hand gesture recognition (Jester) with a streaming TENN - #120

Open
mtorresBrainChip wants to merge 5 commits into
mainfrom
content/HUB-44-jester
Open

mtorresBrainChip wants to merge 5 commits into
mainfrom
content/HUB-44-jester

Conversation

@mtorresBrainChip

Copy link
Copy Markdown
Collaborator

Refs HUB-44

What this adds

akida2/model_zoo/jester: an eval-only example of BrainChip's TENN spatiotemporal model for hand gesture recognition on Jester, on Akida 2. It has no training stage. It evaluates the published float model, reproduces BrainChip's 8-bit post-training quantization and checks it against the published 8-bit model, converts it to Akida, and benchmarks it on the Akida 2 FPGA.

It is the first streaming (temporal) model in the zoo, so the README and notebooks explain how quantization turns the Conv3D model into a buffered one, and how to evaluate a model that sees one frame at a time:

  • reset the buffers per clip;
  • stream one frame per forward call;
  • accumulate the per-frame predictions into a clip decision.

Results

All accuracy figures are on BrainChip's public 541-clip Jester subset, not the full validation split. The README says so wherever they appear.

Variant Clip accuracy Frame accuracy
Float (whole clip) 95.01 % 71.15 %
8-bit PTQ (streaming) 94.64 % 70.52 %
Akida (streaming, software backend) 94.64 % 70.52 %

Per-frame latency on the Akida 2 FPGA (6 nodes, 25 MHz), projected to the 1 GHz AKD2500 target:

Mapping NPs Passes @ 25 MHz (ms) Projected @ 1 GHz (ms)
Minimal 44 2 209.327 5.233
AllNPs 45 2 209.115 5.228
HwPr 96 5 168.036 4.201

NP counts include the input layer's dedicated input processor as one NP, so they are one more than ak_model.summary() reports. On hardware, a clip streams bit-identically to the software backend. Forwarding a clip as one batch also gives the same outputs as one frame per call.

Also in this PR

  • akida2/README.md: a new Video section in the model zoo table (DOMAINS in akida2/update_readme.py), plus a link from the root README.
  • test/test_models.py: the float quantize-sanity test passes batch_size (spatiotemporal TENNs need it to calibrate) and builds its random inputs from the quantized model's input shape (one frame, not a clip).

How it was run

  • Evaluation, on the software backend, from akida2/model_zoo/jester:

    python jester_eval.py --variant float --save-metrics
    python jester_eval.py --variant quantized --save-metrics
    python jester_eval.py --variant akida --save-metrics

    The quantization reproduction (jester_eval.sh) gives the same structure and, after conversion, identical integer weights and thresholds as the published model.

  • Hardware benchmark, on the Akida 2 FPGA:

    python jester_benchmark.py --save-metrics    # about 40 min
    python -m brainchip_utils.notebooks jester_notebook_benchmark.ipynb --timeout 3600   # 1498 s
  • Checks:

    • update_readme.py: a second run leaves no diff;
    • check_content.py --base origin/main: 0 findings;
    • pytest test/test_hardware_utils.py: 3 passed;
    • pytest test/test_models.py -m "not hardware" on the Jester models: 1 passed.

    The Jester hardware tests run in models-hardware.

Not run

  • Evaluation on the full Jester validation split (14,787 clips). The example supports it with -d if you have the dataset in BrainChip's TFRecord layout, but that isn't publicly hosted.

🤖 Generated with Claude Code

mtorresBrainChip and others added 5 commits October 26, 2026 13:25
…eval-only)

Adds an Akida 2 example for BrainChip's published TENN spatiotemporal
Jester model: evaluates the float model on whole clips, reproduces the
8-bit quantization and checks it against the published model, converts
to Akida and evaluates streaming frame by frame. Adds a Video domain to
the Akida 2 landing page and knowledge-base entries on TENN streaming.

The FPGA benchmark has not been run yet: the benchmark keys in
metrics.json are TBD and the benchmark notebook has no outputs.

Refs HUB-44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e pass

Add the Akida 2 FPGA benchmark (per-frame latency in Minimal, AllNps and
HwPr), its plots and the benchmark notebook with hardware outputs. Execute
the evaluation notebook, describe what the mapping modes show, tidy the
prose, and use American spelling throughout the example.

Refs HUB-44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erized_model does

Evaluation and sparsity now forward each frame in its own call on a new model
per clip, instead of forwarding a clip as one batch. Results are unchanged:
per-layer sparsity is identical and the mean is still 56.63%. The README no
longer claims that Akida reads the batch axis as time; the benchmark now
checks that on hardware and reports it, without depending on it.

Drops the knowledge-base and generator-command edits from this branch; they
will follow separately once verified on hardware.

Refs HUB-44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…eaming

Reruns jester_benchmark.py --save-metrics and the benchmark notebook on the
Akida 2 FPGA with the code from 7348c32. The hardware streams a clip
bit-identically to the software backend, and forwarding a clip as one batch
gives the same outputs. Cycle counts moved by less than 0.01%; NP counts,
passes and projected latencies are unchanged. Accuracy and sparsity are not
affected (they come from jester_eval.py).

Drops the model zoo comparison paragraph from the Model Card.

Refs HUB-44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The benchmark table counts the input layer's dedicated input processor as one
NP, so its counts are one more than ak_model.summary() and the BrainChip model
zoo page report (44 against 43 for Minimal).

Refs HUB-44

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sbruers-brainchip
sbruers-brainchip self-requested a review November 6, 2026 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant