feat(eval): add audio classification evaluation and music genre recipes

[ssss141414]

Summary

This contribution adds generalized audio-classification evaluation and refreshes CPU FP32/FP16 recipes for dima806/music_genres_classification. The L2 outcome adds task-family evaluator, configuration, CLI, and regression-test support rather than checkpoint-specific production logic. Goal L3 remains PASS for repaired candidate caefc9ccde629d2596f26ccde41fb5dfdb5092ec through the exact execution/reuse provenance below; the two-row result is functional-smoke evidence only, not representative accuracy or benchmark quality.

Model metadata

What the model does

A Wav2Vec2 music-genre classifier that accepts mono waveform samples at 16 kHz and emits ten logits for the GTZAN genres.

• Evidence/confidence: pinned checkpoint dima806/music_genres_classification@5f71fb1e2c6bedcddb2bfb1e929fc70655780902; its model card identifies GTZAN music genre classification, exact config selects Wav2Vec2ForSequenceClassification with ten id2label entries, exact preprocessor config sets 16,000 Hz and normalization, and the exported ONNX contract is input_values[1,16000] -> logits[1,10] (verified).

Primary user stories

• A music catalog supplies an audio clip to obtain a genre label for recommendation, organization, discovery, or metadata enrichment (verified from the pinned model card).
• A broadcaster, licensing workflow, or researcher supplies recorded music to obtain a coarse genre classification for programming or analysis (verified from the pinned model card).

Supported tasks

• audio-classification across the checkpoint, Transformers, Optimum ONNX, and WinML surfaces. The Hub pipeline tag, Optimum Wav2Vec2 vendor registry, and WinML inspect/config/build resolution agree (verified). Optimum already registers Wav2Vec2 audio classification; WinML adds no vendor-registry alias.

Model architecture

Wav2Vec2ForSequenceClassification
|- Wav2Vec2Model
|  |- Convolutional feature encoder x 7 (total stride 320)
|  |- Feature projection (512 -> 768)
|  |- Positional convolution embedding
|  `- Transformer encoder x 12
|     |- Self-attention (12 heads)
|     `- Feed-forward (768 -> 3072 -> 768, GELU)
|- Linear projector (768 -> 256)
|- Time mean pooling
`- Linear classifier (256 -> 10 logits)

• Source/confidence: exact pinned Wav2Vec2 config, Transformers 4.57.6 Wav2Vec2ForSequenceClassification source, a 94,571,146-parameter trace, and the exported ONNX graph (verified).

Validation and support evidence

Baseline

Baseline was current main commit 8d631f6f1e5db26a04e8c13045280408c6f984dc with winml, version 0.0.1.dev0. Auto-config resolved audio-classification. CPU build passed in 61.2 s (33.7 s export, 16.4 s optimize); baseline perf was 61.63 ms mean, 62.29 ms p50, 64.19 ms p90, 16.23 samples/s, and +62.2 MB RAM. Baseline Eval failed because audio-classification was not registered. The observed --no-optimize behavior is owned separately and is not changed here. Optimum's pinned Wav2Vec2 registry already listed audio-classification, audio-frame-classification, audio-xvector, automatic-speech-recognition, and feature-extraction; WinML added no registry entries.

Goal

• Effort: L2, a generalized task-family implementation plus exact-model recipes.
• Goal ceiling: L3, requiring L0 FP32/FP16 structural proof, L1 CPU perf, L2 numerical parity against pinned PyTorch, and one bounded final-SHA FP32 CPU task-metric smoke.
• Outcome: L2.
• Success definition: both CPU precisions build with their realized graph precision, both record perf, both preserve frozen parity bounds and top-1, and final-SHA FP32 Eval processes two semantically verified rows under explicit duration/window caps and emits accuracy and macro-F1.

Outcome

The highest reached verdict is L3 PASS with full planned CPU FP32/FP16 coverage and no deferred tuples. The shipped Lane B delta contains two refreshed recipes, generalized evaluator/config/CLI registration, and regression tests. Model findings wav2vec2-018 through wav2vec2-022 and methodology finding _meta-115 were published separately in Lane A PR gim-home/ModelKitArtifacts#307, head 837b125a4e60372c2b446fd8003a366645fec95a; those skill/knowledge files are not part of this winml-cli change.

PR #1388 already exists as a draft with the model-scale-by-skill label, but it still points to prior head 95506e515042b10567c4392ada4481df079f2dfd. Repaired candidate caefc9ccde629d2596f26ccde41fb5dfdb5092ec is clean and one commit ahead of the remote branch; this report does not claim it has been pushed or that the PR has been updated.

L0, L1, L2, and Analyze were executed on 4eb6900fa1857a1feaa565a3774e054e3a2f06d0. They were first REUSE-REBIND evidence for 95506e515042b10567c4392ada4481df079f2dfd, not relabeled final-SHA executions; L3 and its then-current quality gates were run fresh on 95506e515042b10567c4392ada4481df079f2dfd. For repaired candidate caefc9ccde629d2596f26ccde41fb5dfdb5092ec, L0-L3, Analyze, Models, Optim, mypy, Remaining, and license evidence are again REUSE-REBIND, preserving those original execution SHAs. The exact 95506e515042b10567c4392ada4481df079f2dfd..caefc9ccde629d2596f26ccde41fb5dfdb5092ec diff changes only tests/unit/eval/test_audio_classification_evaluator.py: it removes the mixed ONNX from-import and qualifies the same TensorProto and helper objects through onnx, with identical fixture graph operations, arguments, opset, IR, save behavior, and no production, recipe, dependency, artifact, or runtime-path change. Fresh repaired-head checks found no from onnx import, passed the import/runtime probe, passed 34 focused tests, passed Ruff, and passed Commands with 3726 tests and 9 skips.

Per-EP/device/precision results and Functional smoke Eval

Tier	EP / device	Precision	Verdict	Mean	p50	Throughput	RAM delta
L0	CPU / cpu	fp32	PASS	-	-	-	-
L0	CPU / cpu	fp16	PASS	-	-	-	-
L1	CPU / cpu	fp32	PASS	49.85 ms	49.94 ms	20.06 samples/s	+62.5 MB
L1	CPU / cpu	fp16	PASS	59.48 ms	59.17 ms	16.81 samples/s	+41.3 MB

Both L1 rows used 3 warmups and 10 measured iterations. FP16 was slower than FP32 in this short CPU run; no speed threshold applies.

L0 structural validation. Both artifacts passed ONNX load/checker at IR 8 and opset 17, with FP32 graph I/O input_values[1,16000] FLOAT -> logits[1,10] FLOAT. FP32 has 385 nodes, 276 initializers (216 FLOAT, 60 INT64), and 378,438,918 total bytes (153,862-byte model plus 378,285,056-byte external data). FP16 has 387 nodes, 276 initializers (216 FLOAT16, 60 INT64), and 189,296,375 total bytes (154,359-byte model plus 189,142,016-byte external data). This proves true FP16 initializer conversion while preserving FP32 I/O; both use eager attention and local external data.

L2 numerical parity. One real 1,323,632-byte WAV was decoded from mono 22,050 Hz / 661,794 frames, resampled to 16,000 Hz, reduced to 16,000 normalized samples, and supplied as input_values[1,16000] float32 to pinned PyTorch and ONNX. Frozen FP32 bounds were cosine >= 0.9999 and maximum absolute error <= 0.01; observed cosine was 1.0, maximum absolute error 1.0371208190917969e-05, and reference/candidate top-1 were both 6. Frozen FP16 bounds were cosine >= 0.999 and maximum absolute error <= 0.05; observed cosine was 0.9999998211860657, maximum absolute error 0.0035549402236938477, and reference/candidate top-1 were both 6.

Functional smoke Eval. This L3 run executed on 95506e515042b10567c4392ada4481df079f2dfd and is REUSE-REBIND evidence for repaired candidate caefc9ccde629d2596f26ccde41fb5dfdb5092ec; it passed using FP32 ONNX Runtime CPU and pinned danilotpnta/GTZAN_genre_classification@77564793c44e244eaff98a267bb7ba7cc1bda0b1, config/split gtzan/train. Exactly 2 samples were requested, 101 eligible rows were examined, 2 were selected/decoded/processed, 4 inference windows ran, 2 samples were truncated, and 0 were rejected. The authoritative scalar labels mapped exactly and case-sensitively: blues -> 3 and classical -> 5. Both embedded PCM_16 WAVs were mono 22,050 Hz with 661,794 decoded float32 frames; each was resampled to 16,000 Hz, capped at 2.0 seconds / 32,000 samples, and split into exactly two 16,000-sample windows. Mean logits were aggregated once per utterance before argmax and pinned id2label decoding. Predictions were blues -> jazz and classical -> classical; accuracy was 0.5 and macro-F1 was 0.5. This proves bounded end-to-end operability only and is not representative accuracy or benchmark quality.

The evaluator supports raw audio bytes and ordinary paths, preserves virtual archive paths for xopen, and decodes with SoundFile without requiring TorchCodec or FFmpeg. It converts decoded float32 audio to mono, resamples to the checkpoint rate, applies optional duration and static-window caps, requires one static audio input and one rank-2 class-logits output, maps scalar labels exactly through configured metadata, and computes utterance accuracy/macro-F1 after mean-logit aggregation.

Fresh repaired-head evidence on caefc9ccde629d2596f26ccde41fb5dfdb5092ec: ONNX import/runtime probe PASS with no from onnx import; focused audio regressions 34 passed, 1 warning; touched-file Ruff PASS; Commands 3726 passed, 9 skipped, 2 warnings. Reused evidence retains its original execution SHA: invalidated Eval/schema/recipe tests 173 passed, 1 warning, full Ruff and license headers PASS, L3, Analyze 1529 passed, 45 skipped, Models 1541 passed, 7 skipped, 1 xfailed, Optim 876 passed, 16 skipped, 1 xfailed, and Remaining 964 passed, 2 skipped, 1 deselected were established on 95506e515042b10567c4392ada4481df079f2dfd or its explicitly cited source chain. Mypy remains provenance-bound at 441 files, 0 issues on baseline and 442 files, 0 issues on 95506e515042b10567c4392ada4481df079f2dfd; it was not rerun or relabeled as a caefc9ccde629d2596f26ccde41fb5dfdb5092ec execution.

Delta

The base-to-candidate range is 8d631f6f1e5db26a04e8c13045280408c6f984dc..caefc9ccde629d2596f26ccde41fb5dfdb5092ec: 11 files changed, 2089 insertions(+), 3 deletions(-) across five commits. examples/recipes/README.md is untouched.

File	Exact delta
examples/recipes/dima806_music_genres_classification/cpu/cpu/audio-classification_fp32_config.json	Refreshes /export/compatibility/transformers_attention from absent to "eager"; retains FP32 recipe semantics.
examples/recipes/dima806_music_genres_classification/cpu/cpu/audio-classification_fp16_config.json	Refreshes /export/compatibility/transformers_attention from absent to "eager"; retains FP16 recipe semantics.
src/winml/modelkit/commands/eval.py	Plumbs additive audio duration configuration through Eval CLI handling.
src/winml/modelkit/eval/__init__.py	Adds lazy export of the audio-classification evaluator.
src/winml/modelkit/eval/audio_classification_evaluator.py	Adds generalized bytes/path/virtual-path decoding, SoundFile preprocessing, mono/resampling, duration/window bounds, static-shape validation, exact label mapping, mean-logit decoding, accuracy, and macro-F1.
src/winml/modelkit/eval/config.py	Adds optional finite positive DatasetConfig.max_duration_seconds with null/unbounded default and label-mapping round-trip.
src/winml/modelkit/eval/evaluate.py	Registers the generalized audio-classification evaluator.
src/winml/modelkit/utils/eval_utils.py	Adds the audio-classification task schema and dataset/config fields.
tests/unit/eval/test_audio_classification_evaluator.py	Adds focused coverage for raw bytes, paths and virtual paths, decoding/preprocessing, caps, mappings, aggregation/metrics, and invalid model shapes; the reviewer repair normalizes ONNX imports by using onnx.TensorProto and onnx.helper exclusively.
tests/unit/eval/test_eval.py	Adds registry/config/CLI integration regressions while preserving existing tasks.
tests/unit/recipes/test_cpu_recipes.py	Refreshes exact FP32/FP16 recipe expectations.

Bug fix explanation.

1. Symptom/minimal trigger: generalized audio Eval could reject or backend-decode valid streaming rows before SoundFile saw their raw bytes/virtual path, and the replacement public dataset's authoritative scalar Value('string') genre labels were rejected before configured mapping.
2. Root cause: the old flow did not preserve every pre-encoding bytes/path representation and restricted accepted target feature shapes too early, ahead of exact label mapping and bounded sample selection.
3. Changed symbols/mechanism: WinMLAudioClassificationEvaluator, _AudioModelAdapter, DatasetConfig, Eval registry/schema, and CLI plumbing now preserve embedded bytes or xopen-readable virtual paths, decode through SoundFile, normalize audio shape/rate, validate static model contracts, retain scalar string keys for explicit mapping, aggregate window logits, and emit accuracy/macro-F1.
4. General rule: dispatch is derived from canonical task, row media representation, model input/output metadata, feature extractor settings, and caller-supplied label mapping. The production delta adds no checkpoint ID or dataset literal.
5. Compatibility/blast radius: existing Eval tasks and CLI defaults remain unchanged; duration is unbounded unless explicitly capped; explicit mapping is required rather than inferred from paths or ordering; FP32/FP16 recipes preserve FP32 I/O and precision semantics. Unsupported multi-input, dynamic audio-shape, multi-head, and non-rank-2 logits models fail clearly instead of being guessed.
6. Regression evidence: 95506e515042b10567c4392ada4481df079f2dfd established the pinned two-row smoke, 173 invalidated schema/recipe tests, L0-L3, Analyze, full Ruff/license, mypy, and the non-Commands partitions through the stated provenance chain. caefc9ccde629d2596f26ccde41fb5dfdb5092ec freshly passed the import/runtime probe with no from onnx import, 34 focused tests, touched-file Ruff, and Commands at 3726 passed / 9 skipped. The exact test-only alias-qualification diff has no behavior impact, so the prior model/runtime evidence is REUSE-REBIND rather than relabeled as freshly executed.

The class-wide code fix is recipe-free and metadata-driven; the recipes only record exact checkpoint/precision compatibility. Reducibility is consistent with the L2 charter.

Analyze summary - component level and op level

Analyze was ANALYZE-PARTIAL-SUCCESS for both artifacts; each command exited 1 because some requested EPs had no rule data. This is static rule analysis, not runtime execution or an accelerator support claim.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32	convolutional feature encoder; feature projection; 12x encoder; projector; mean pooling; classifier	246 mapped, 139 partial/unmapped of 385; partial confidence	No operator-rule issue; optimization removed scope from 139 fused/support nodes, which remain explicitly unmapped rather than guessed.
fp16	same regions	246 mapped, 141 partial/unmapped of 387; partial confidence	Same mapping gap, including two graph-boundary Cast nodes.

Op-level summary

Artifact	Graph	Dominant ops	Static EP roll-up
fp32	385 ops / 14 types	Reshape 126; Gemm 75; Transpose 51; Add 26; LayerNormalization 26	TensorRT RTX, QNN GPU, and OpenVINO GPU rules classify all 385 supported.
fp16	387 ops / 15 types	Reshape 126; Gemm 75; Transpose 51; Add 26; LayerNormalization 26; Cast 2	TensorRT RTX, QNN GPU, and OpenVINO GPU rules classify all 387 supported.

CUDA, MIGraphX, legacy TensorRT, and DirectML had no loaded rule data and were skipped/unknown; that static limitation caused the nonzero Analyze status. No runtime inference was performed on those EPs.

Reproduce commands

The commands below preserve the Tester invocations with portable variables in place of machine-local interpreter, model-cache, and output paths. label-map.json contains { "blues": 3, "classical": 5 }.

$PYTHON = "python"
$MODEL_ID = "dima806/music_genres_classification"
$REVISION = "5f71fb1e2c6bedcddb2bfb1e929fc70655780902"
$OUT = Join-Path $PWD "model12-output"
New-Item -ItemType Directory -Force $OUT | Out-Null

& $PYTHON -m winml.modelkit build -c examples/recipes/dima806_music_genres_classification/cpu/cpu/audio-classification_fp32_config.json -m $MODEL_ID -o "$OUT/fp32"
& $PYTHON -m winml.modelkit build -c examples/recipes/dima806_music_genres_classification/cpu/cpu/audio-classification_fp16_config.json -m $MODEL_ID -o "$OUT/fp16"

& $PYTHON -m winml.modelkit perf -m "$OUT/fp32/model.onnx" --ep cpu --device cpu --warmup 3 --iterations 10
& $PYTHON -m winml.modelkit perf -m "$OUT/fp16/model.onnx" --ep cpu --device cpu --warmup 3 --iterations 10

& $PYTHON -m winml.modelkit analyze --model "$OUT/fp32/model.onnx" --ep all --output "$OUT/analysis-fp32.json"
& $PYTHON -m winml.modelkit analyze --model "$OUT/fp16/model.onnx" --ep all --output "$OUT/analysis-fp16.json"

'{ "blues": 3, "classical": 5 }' | Set-Content -Encoding utf8 "$OUT/label-map.json"
& $PYTHON -m winml.modelkit eval -m "$OUT/fp32/model.onnx" --model-id $MODEL_ID --task audio-classification --dataset danilotpnta/GTZAN_genre_classification --dataset-name gtzan --dataset-revision 77564793c44e244eaff98a267bb7ba7cc1bda0b1 --split train --samples 2 --no-shuffle --streaming --column label_column=genre --label-mapping "$OUT/label-map.json" --max-duration-seconds 2.0 --ep cpu --device cpu --skip-build --format json -o "$OUT/l3-eval.json"

& $PYTHON -m pytest tests/unit/eval/test_audio_classification_evaluator.py tests/unit/eval/test_eval.py tests/unit/recipes/test_cpu_recipes.py -q

The Tester used Python 3.11.15, uv 0.12.5, Transformers 4.57.6, ONNX 1.18.0, onnxruntime-windowsml 1.27.1.202607110137, Datasets 5.0.1, SoundFile 0.14.0, and Torch 2.14.0 with lock digest 8353285ee649c5df37f15e8129c1bcc3d24f178e38c88dafcc18b95a16ee86d8.

[ssss141414]

REQUEST_CHANGES

One blocking finding:

• Producer: tests/unit/eval/test_audio_classification_evaluator.py:16 imports ONNX using both import onnx and from onnx import TensorProto, helper. GitHub Advanced Security opened an unresolved exact-head CodeQL thread (Module is imported with 'import' and 'import from'). I independently reproduced the claim from candidate 95506e515042b10567c4392ada4481df079f2dfd. Normalize the imports in this test file and push a new candidate. Do not change production behavior for this finding.

After the Producer repair, Tester must rerun the focused audio/eval/recipe tests, full Ruff, and the canonical commands partition. Explainer must refresh candidate/report binding if the head changes, then a fresh Reviewer must re-enumerate all comments/threads and wait for every exact-head GitHub check to complete.

The rest of the review is clean: exact candidate/tree/base and sealed title/body; five authoritative and seven historical manifests rehashed; implementation, recipes, L0/L1/L2/L3/Analyze evidence, REUSE-REBIND, and Lane A PR #307 binding verified; independent 110 passed focused, full Ruff PASS, mypy 442 files clean, and 3726 passed / 9 skipped in commands. GitHub CI currently shows all nine checks COMPLETED/SUCCESS, but successful CI does not override an unresolved review thread.

Review surface: 1 line comment, 1 review thread, 1 open thread. PR remains OPEN draft with model-scale-by-skill. Models13+ remain QUEUED.

This is an ordinary conversation comment, not a GitHub Review state. Do not ready or merge the PR.

[ssss141414]

APPROVE

No blocking findings.

Reviewed exact candidate caefc9ccde629d2596f26ccde41fb5dfdb5092ec (tree a53a942cb699716c836dc8749195e9cb6a1903c8, parent 95506e515042b10567c4392ada4481df079f2dfd) against base 8d631f6f1e5db26a04e8c13045280408c6f984dc and the complete immutable Model12 lineage. The 11-file Outcome-L2 change is correctly scoped: generalized audio-classification evaluator/config/CLI/schema support, two exact CPU recipes, and regression tests. The implementation uses metadata-driven dispatch; raw bytes, ordinary paths, and virtual archive paths decode through the declared SoundFile path; scalar string, ClassLabel, and multi-label semantics remain explicit; resampling, duration/static-window caps, mean-logit utterance aggregation, exact configurable label mapping, accuracy/macro-F1, and malformed-contract handling are sound. No target model/dataset hardcoding, scratch leakage, README change, or public-default regression was found.

The repair from 95506e515042b10567c4392ada4481df079f2dfd changes only tests/unit/eval/test_audio_classification_evaluator.py: it removes from onnx import ... and qualifies the same runtime objects through onnx. No from onnx import remains, runtime identity checks return True True, and there is no behavior change.

Frozen evidence was independently rehashed and sampled. L0-L2 and Analyze retain original execution SHA 4eb6900fa1857a1feaa565a3774e054e3a2f06d0; L3 retains execution SHA 95506e515042b10567c4392ada4481df079f2dfd; repair-focused quality is fresh at caefc9ccde629d2596f26ccde41fb5dfdb5092ec. FP16 has 216 FLOAT16 initializers and 189,142,016-byte external data versus FP32's 378,285,056 bytes. L1 is 49.85 ms / 20.06 samples/s / +62.5 MB FP32 and 59.48 ms / 16.81 samples/s / +41.3 MB FP16. L2 cosine/max-absolute-error is 1.0 / 1.04e-5 FP32 and 0.99999982 / 0.003555 FP16. The pinned two-row GTZAN smoke decodes real WAV bytes, processes four bounded windows, and reports accuracy/macro-F1 0.5/0.5 as operability only. Component/op Analyze preserves mapping gaps and complete requested-EP classifications. The formal REUSE-REBIND is valid because the repair is test-import-only.

Independent exact-head validation: focused 34 passed; full Ruff passed; mypy passed for 442 source files; license headers passed; canonical affected commands partition passed with 3726 passed, 9 skipped. Lane A commits 59a0b133 and 837b125a exist; the exact four-file delta is contained by origin/ssss141414/learner-meta-115-model12; sealed authenticated evidence binds PR #307 and findings wav2vec2-018..022 plus _meta-115.

PR #1388 is OPEN draft by ssss141414, base main, exact branch/head, mergeable, exact sealed title/body (normalized length 18425), and labeled model-scale-by-skill. Enumerated 2 line comments, 1 conversation comments, 2 reviews, 1 review thread, and 0 open threads. The sole prior blocker discussion_r3918559651 has repair reply discussion_r3919208836; thread PRRT_kwDORaJc5s6erYfa is resolved.

Exact-head GitHub checks:

• Analyze (Python): COMPLETED/SUCCESS
• CodeQL: COMPLETED/SUCCESS
• license/cla: COMPLETED/SUCCESS
• lint: COMPLETED/SUCCESS
• test (analyze): COMPLETED/SUCCESS
• test (commands): COMPLETED/SUCCESS
• test (models): COMPLETED/SUCCESS
• test (optim): COMPLETED/SUCCESS
• test (remaining): COMPLETED/SUCCESS

Coverage is full: verified CPU/cpu/fp32 and CPU/cpu/fp16; deferred tuples: none.

Disposition: terminal skill-level APPROVE. Keep the PR draft. This is an ordinary conversation comment, not a GitHub Review state.