feat(eval): add audio classification evaluation and MMS language ID recipes

[ssss141414]

Summary

This contribution adds generalized audio-classification evaluator, configuration, CLI, and schema support and refreshes CPU FP32/FP16 recipes for facebook/mms-lid-256. The L2 outcome is independently mergeable and uses model metadata plus explicit label mappings rather than Model12 recipes, checkpoint hardcoding, or README changes. Goal L3 passed on candidate 2e0516d378111677492758aa522ff8135b13a990; its two-row result is functional-smoke evidence only and reports 0 accuracy and 0 macro-F1, not model-quality success.

Model metadata

What the model does

facebook/mms-lid-256 is a 256-class spoken-language identification model. It accepts mono 16 kHz raw speech waveform samples and returns one logits vector whose argmax maps to an ISO 639-3 language code.

• Evidence/confidence: Hugging Face model card and config at pinned revision edc73fd00996e671dfc59d16436a29b12b10588a; architectures=Wav2Vec2ForSequenceClassification, a 256-entry id2label, and sampling_rate=16000 (verified).

Primary user stories

• A user supplies a speech recording to obtain its predicted spoken language for language routing or content indexing (verified from the pinned model card and audio-classification pipeline tag).
• An application supplies a short speech window to choose an appropriate downstream recognizer or localization workflow (mapped from the pinned language-identification task, 256 ISO 639-3 outputs, and fixed WinML waveform contract).

Supported tasks

• audio-classification across the checkpoint, Transformers, Optimum ONNX, and WinML surfaces. The Hub pipeline tag, Wav2Vec2ForSequenceClassification, Optimum vendor registry, and WinML inspect resolution to AutoModelForAudioClassification plus Wav2Vec2OnnxConfig agree (verified). Optimum already registers Wav2Vec2 audio classification; WinML adds no vendor-registry alias.

Model architecture

Wav2Vec2ForSequenceClassification (966,222,720 parameters)
|-- Wav2Vec2Model
|   |-- Feature encoder: 7 temporal Conv + LayerNorm + GELU stages (channels 512)
|   |-- Feature projection: 512 -> 1280
|   |-- Positional convolution embedding
|   `-- Stable-LayerNorm encoder stack x 48
|       |-- Multi-head self-attention (16 heads)
|       |-- Adapter layer
|       `-- Feed-forward (1280 -> 5120 -> 1280, GELU)
|-- Frame projector: 1280 -> 1024
|-- Temporal mean pooling
`-- Language classifier: 1024 -> 256 logits

• Source/confidence: pinned Transformers class source and config, plus the exported input_values[1,16000] -> logits[1,256] ONNX contract (verified).

Validation and support evidence

Baseline

Baseline was main commit 8d631f6f1e5db26a04e8c13045280408c6f984dc with WinML 0.3.0. Inspect and auto-config resolved Wav2Vec2 audio-classification; recipe-free CPU FP32 build passed in 428.1 s with static FP32 input_values[1,16000] -> logits[1,256]. A bounded three-iteration CPU run measured 312.12 ms mean, 311.27 ms p50, 3.20 samples/s, and +111.5 MB RAM. Pinned PyTorch/ONNX parity passed at cosine 0.9999999999859412, max absolute error 3.8623809814453125e-05, and matching top-1 96/nno. Eval was CLI-blocked before dataset loading because audio-classification was not registered. Baseline Analyze produced a complete 2,001-op/15-type report with partial static support semantics and exit 1.

The baseline was reused because the prior and current main SHAs were identical, so the reviewed diff was empty and no dependency surface changed. Optimum's vendor registry already listed audio-classification, audio-frame-classification, audio-xvector, automatic-speech-recognition, and feature-extraction; WinML added no registry entries.

Goal

• Effort: L2, generalized task-family implementation plus exact-model recipe refresh.
• Goal ceiling: L3, requiring L0 FP32/FP16 structural proof, L1 CPU performance and memory, L2 numerical parity against pinned PyTorch, and one bounded final-candidate FP32 CPU task-metric smoke.
• Outcome: L2.
• Success definition: both CPU precisions build with realized graph precision and exact static I/O, both record performance, both preserve frozen parity bounds and top-1, and FP32 Eval processes two semantically verified rows under one-second/one-window caps and emits accuracy and macro-F1.

Outcome

The highest reached verdict is L3 PASS-FUNCTIONAL-SMOKE with full planned CPU FP32/FP16 coverage and no deferred tuples. The Lane B delta contains two refreshed MMS-LID recipes, generalized evaluator/config/CLI/schema support, and regression tests. Findings wav2vec2-023 through wav2vec2-026 and methodology finding _meta-116 are separate Lane A work in gim-home/ModelKitArtifacts#308, head 2d581008e92af2dea2664b69060fb78b6338feb7; no skill or knowledge file is in this winml-cli delta.

This branch is based directly on main, not stacked on Model12 PR microsoft/winml-cli#1388. That PR is currently draft and unmerged, and both branches independently carry compatible generalized audio-classification support. This contribution requires neither Model12 recipes nor fixtures and can merge independently. The overlapping shared evaluator changes will need normal conflict resolution or deduplication according to merge order; if #1388 merges before this review, this branch must be rebased onto updated main, duplicate shared changes dropped, and impacted validation rerun before merge.

Per-EP/device/precision results and Functional smoke Eval

Tier	EP / device	Precision	Verdict	Mean	Median	Throughput	RSS before -> measured (delta)
L0	CPUExecutionProvider / cpu	fp32	PASS	-	-	-	-
L0	CPUExecutionProvider / cpu	fp16	PASS	-	-	-	-
L1	CPUExecutionProvider / cpu	fp32	PASS	305.474 ms	297.984 ms	3.2736 samples/s	49,852,416 -> 3,983,691,776 bytes (+3,933,839,360)
L1	CPUExecutionProvider / cpu	fp16	PASS	352.231 ms	359.171 ms	2.8390 samples/s	49,934,336 -> 4,003,487,744 bytes (+3,953,553,408)

Each L1 row used a fresh child process, the same deterministic [1,16000] waveform, 2 warmups, and 5 measured iterations. Exact RSS snapshots were FP32 49,852,416 before load, 3,866,636,288 after load, and 3,983,691,776 after warmup/measurement; FP16 was 49,934,336, 3,957,334,016, and 4,003,487,744 bytes respectively. FP16 was slower and used slightly more process RSS in this short CPU run; no speed or memory-improvement threshold applies.

L0 structural validation. FP32 is exactly 3,864,865,108 bytes with 908 FLOAT initializers. FP16 is exactly 1,932,787,081 bytes with 908 FLOAT16 initializers while preserving FP32 graph/runtime I/O. Both have static input_values[1,16000] -> logits[1,256], finite output, and top-1 96/nno. Both use ONNX external data; the large-model FP16 conversion logged that shape inference could not serialize the external-data model and completed through the explicit shape-inference-disabled fallback.

L2 numerical parity. The deterministic one-second, 16,000-sample FLEURS waveform produced PyTorch logits [1,256]. FP32 passed with cosine 0.9999999999983611, max absolute error 1.0102987e-05, and matching PyTorch/ONNX top-1 96/nno. FP16 passed with cosine 0.9999992021219782, max absolute error 0.0077041388, and matching top-1 96/nno.

Functional smoke Eval. Candidate 2e0516d378111677492758aa522ff8135b13a990 passed using FP32 ONNX Runtime CPU and pinned google/fleurs@70bb2e84b976b7e960aa89f1c648e09c59f894dd, config de_de, split test, streaming with no shuffle. The authoritative label bridge was exactly FLEURS 17/de_de/German -> 55/deu; no locale inference was used. Exactly 2 samples were requested, eligible, selected, decoded, and processed, producing 2 total inference windows. Both utterances were truncated to one second / 16,000 samples and capped at one window each. Targets were deu, deu; predictions were cym, nno; accuracy was 0.0 and macro-F1 was 0.0. This is emphatically a bounded functional smoke proving evaluator and model-path operability only. It is non-representative and is not evidence of benchmark quality or model-quality success.

The former blocker was the absence of an audio-classification Eval task. The generalized evaluator now supplies explicit dataset/schema configuration, SoundFile raw-audio decoding, mono/resampling and static-window preprocessing, exact configurable labels, utterance-level mean-logit aggregation, prediction decoding, accounting, accuracy, and macro-F1.

Quality evidence. Focused tests: 39 passed, 1 warning. Eval/recipe regressions: 704 passed, 1 warning. Analyze: 1529 passed, 45 skipped. Models/loader/datasets/export: 1541 passed, 7 skipped, 1 xfailed. Optim: 876 passed, 16 skipped, 1 xfailed. Commands/config/build/compiler/session/eval: 3727 passed, 9 skipped, 2 warnings. Mypy was clean on 441 source files at the baseline and 442 source files on the candidate. Touched Ruff, recipe JSON/schema validation, and checkpoint/dataset license checks passed.

The disjoint remaining partition must retain its raw result as FAIL: 6 failed, 2536 passed, 9 skipped, 1 xfailed, 1 warning. Tester reran the exact six selectors against parent and candidate with the same interpreter/environment and reproduced all six identically, so candidate-delta grading is PASS-WITH-SIX-BASELINE-EXCLUSIONS; the raw run is not relabeled as passing:

• test_no_direct_ep_device_imports_in_src: identical existing export/policy.py direct-import policy violation.
• test_no_inline_ep_device_mapping_literals_in_src: identical existing perf.py and openvino_monitor.py inline mappings.
• test_demo_locks_server_selection_to_document_origin: identical host failure because Node could not be resolved.
• test_demo_preserves_mount_path_in_server_url: identical host failure because Node could not be resolved.
• test_uv_lock_does_not_include_cuda_accelerator_packages: identical checkout failure because both trees lack uv.lock.
• test_uv_lock_records_direct_project_dependencies: identical checkout failure because both trees lack uv.lock.

Delta

The base-to-candidate range is 8d631f6f1e5db26a04e8c13045280408c6f984dc..2e0516d378111677492758aa522ff8135b13a990: 11 files, 2,163 insertions, and 3 deletions. examples/recipes/README.md is untouched; no Model12 recipe, fixture, model ID, or dataset ID is hardcoded into production evaluator logic.

File	Exact delta
examples/recipes/facebook_mms-lid-256/cpu/cpu/audio-classification_fp16_config.json	Parent /export/compatibility: null -> { "transformers_attention": "eager" }; relative to auto-config, adds /quant with mode=fp16, fp16_keep_io_types=true, and the retained recipe quantization fields.
examples/recipes/facebook_mms-lid-256/cpu/cpu/audio-classification_fp32_config.json	Parent /export/compatibility: null -> { "transformers_attention": "eager" }; otherwise identical to the Planner starting auto-config.
src/winml/modelkit/commands/eval.py	Plumbs additive audio-classification dataset and duration configuration through Eval CLI handling.
src/winml/modelkit/eval/__init__.py	Exports the generalized audio-classification evaluator.
src/winml/modelkit/eval/audio_classification_evaluator.py	Adds raw bytes/path/virtual-path SoundFile decoding, mono/resampling, duration/window bounds, static model validation, exact scalar/multi-label mapping, accounting, mean-logit decoding, accuracy, and macro-F1.
src/winml/modelkit/eval/config.py	Adds optional finite positive DatasetConfig.max_duration_seconds and label-mapping configuration.
src/winml/modelkit/eval/evaluate.py	Registers audio-classification evaluator dispatch.
src/winml/modelkit/utils/eval_utils.py	Adds the audio-classification task schema and dataset/config fields.
tests/unit/eval/test_audio_classification_evaluator.py	Covers raw bytes, ordinary/virtual paths, preprocessing, caps, exact mappings, aggregation/metrics, accounting, and invalid model contracts.
tests/unit/eval/test_eval.py	Adds registry, schema, configuration, and CLI integration regressions while preserving existing tasks.
tests/unit/recipes/test_cpu_recipes.py	Refreshes exact MMS-LID FP32/FP16 recipe expectations.

Bug fix explanation.

1. Symptom/minimal trigger: winml eval --task audio-classification failed before dataset loading because the task had no registry/schema/evaluator path; valid streaming audio and external dataset labels therefore could not reach ONNX inference and task metrics.
2. Root cause: the shared Eval stack had no audio-classification dispatch, raw-audio preprocessing, fixed-window utterance aggregation, or exact dataset-to-model label bridge.
3. Changed symbols/mechanism: WinMLAudioClassificationEvaluator, _AudioModelAdapter, DatasetConfig.max_duration_seconds, _EVALUATOR_REGISTRY, and TASK_SCHEMAS now preserve/decode supported audio representations, normalize shape/rate, enforce static model contracts and caps, map labels explicitly, aggregate logits once per utterance, decode predictions, and emit accounting plus metrics.
4. General rule: behavior is selected by canonical task, model input/output metadata, feature-extractor settings, row media representation, and caller-supplied exact mapping. No checkpoint or dataset literal selects production behavior.
5. Compatibility/blast radius: existing evaluator tasks/defaults remain unchanged; duration remains unbounded unless requested; explicit mapping prevents inferred label ordering; unsupported model contracts fail clearly. Existing MMS recipes retain precision and FP32 I/O semantics. The branch has compatible overlap with draft feat(eval): add audio classification evaluation and music genre recipes #1388 but no merge dependency on it.
6. Regression evidence: focused 39, Eval/recipes 704, all named canonical partitions, Ruff/schema/recipe/license checks, and baseline/candidate mypy passed; the exact six unrelated remaining failures reproduced on both parent and candidate. Final candidate L0-L3 and Analyze evidence passed under the limitations reported here.

The class-wide evaluator fix is metadata/configuration-driven and recipe-free; the two recipes record exact checkpoint/precision compatibility. Reducibility is consistent with the L2 charter.

Analyze summary - component level and op level

Analyze completed with PASS-PARTIAL-COMPONENT-MAPPING and CLI exit 1 partial-support semantics. These are static rule and graph-name analyses, not runtime execution or accelerator support claims.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable findings
fp32	feature encoder/projection; positional convolution; 48 encoder layers; frame projector; temporal mean pool; classifier	1,189 mapped, 686 unmapped of 1,875; partial confidence	Optimizer-generated names without a unique Planner scope remain explicitly unmapped.
fp16	same mapped regions	1,189 mapped, 688 unmapped of 1,877; partial confidence	Same mapping gap plus two unmapped precision-boundary Cast nodes.

Op-level summary

Artifact	Graph	Dominant ops	Static rule roll-up
fp32 analyzed artifact	1,875 ops / 14 types	Reshape 676; Gemm 387; Transpose 207; LayerNormalization 153; Add 145	NvTensorRTRTX GPU, QNN NPU/GPU, and OpenVINO NPU/GPU/CPU rules classify all 14 types supported.

The supported rule rows each map all 1,875 operators. CUDA GPU, MIGraphX GPU, legacy TensorRT GPU, DML GPU, CPU, and VitisAI NPU had no loaded rule classifications; they are not runtime failures or support claims. FP16 component analysis has 1,877 graph nodes because of its two boundary Casts, while the tester-owned op-level rule report is the 1,875-op FP32 analyzed artifact.

Reproduce commands

The commands below normalize the tester invocations to repository-relative paths and portable variables. The checkout must be candidate 2e0516d378111677492758aa522ff8135b13a990, and $MODEL must resolve pinned checkpoint revision edc73fd00996e671dfc59d16436a29b12b10588a. label-map.json contains the exact FLEURS-to-model bridge { "de_de": 55 }.

$PYTHON = "python"
$MODEL = "facebook/mms-lid-256"
$REVISION = "edc73fd00996e671dfc59d16436a29b12b10588a"
$OUT = Join-Path $PWD "model13-output"
New-Item -ItemType Directory -Force $OUT | Out-Null

& $PYTHON -m winml.modelkit build -c examples/recipes/facebook_mms-lid-256/cpu/cpu/audio-classification_fp32_config.json -m $MODEL -o "$OUT/fp32" --rebuild
& $PYTHON -m winml.modelkit build -c examples/recipes/facebook_mms-lid-256/cpu/cpu/audio-classification_fp16_config.json -m $MODEL -o "$OUT/fp16" --precision fp16 --rebuild

& $PYTHON -m winml.modelkit perf -m "$OUT/fp32/model.onnx" --ep cpu --device cpu --warmup 2 --iterations 5
& $PYTHON -m winml.modelkit perf -m "$OUT/fp16/model.onnx" --ep cpu --device cpu --warmup 2 --iterations 5

& $PYTHON -m winml.modelkit analyze --model "$OUT/fp32/model.onnx" --ep all --device all --output "$OUT/analyze-fp32.json" --overwrite --no-color

'{ "de_de": 55 }' | Set-Content -Encoding utf8 "$OUT/label-map.json"
& $PYTHON -m winml.modelkit eval -m "$OUT/fp32/model.onnx" --model-id $MODEL --task audio-classification --dataset google/fleurs --dataset-name de_de --dataset-revision 70bb2e84b976b7e960aa89f1c648e09c59f894dd --split test --samples 2 --no-shuffle --streaming --max-duration-seconds 1 --column input_column=audio --column label_column=lang_id --label-mapping "$OUT/label-map.json" --ep cpu --device cpu -o "$OUT/l3-eval.json" --overwrite --no-color

& $PYTHON -m pytest tests/unit/eval/test_audio_classification_evaluator.py tests/unit/recipes/test_cpu_recipes.py --tb=short --no-cov -q --color=no
& $PYTHON -m pytest tests/unit/eval tests/unit/recipes/test_cpu_recipes.py --tb=short --no-cov -q --color=no
& $PYTHON -m mypy -p winml.modelkit

Portable environment provenance: Windows, Python 3.11.15, PyArrow 23.0.1, Datasets 5.0.1, pytest 9.1.1, Ruff 0.16.5, mypy 2.3.1, Transformers 4.57.6, and ONNX Runtime 1.27.1. The test environment was a private dependency-compatible copy used to avoid TLS acquisition failure, with candidate source bound explicitly; no Model12 artifacts or results were reused.

[ssss141414]

APPROVE

Independent Reviewer verdict for facebook/mms-lid-256 at exact candidate 2e0516d378111677492758aa522ff8135b13a990.

• PR state: OPEN draft feat(eval): add audio classification evaluation and MMS language ID recipes #1389, author ssss141414, base/main 8d631f6f, tree 4f74d62e..., label model-scale-by-skill, mergeable, exact 18,528-character body, and exact 11-file delta.
• Scope/design: generalized metadata/config-driven audio-classification evaluator plus MMS FP32/FP16 recipes and regression tests; no Model12 model/dataset hardcoding, README change, scratch leakage, or --no-optimize change. Bytes/path/virtual xopen decoding, ClassLabel preservation, mono/resampling, bounded static windows, utterance mean-logit aggregation, exact label maps, metrics/accounting, and static I/O checks were reviewed.
• Goal evidence: L0-L3 PASS for CPU fp32/fp16, including exact [1,16000] -> [1,256], 3,864,865,108-byte FLOAT and 1,932,787,081-byte FLOAT16 artifacts, independent CLI perf within 6.25%/2.29% of frozen means, exact L2 parity, and pinned two-row FLEURS functional smoke with 0/0 metrics explicitly not a quality claim.
• Analyze: 1,875 ops / 14 types; 1,189 mapped nodes; 686 fp32 and 688 fp16 unmapped nodes, including two fp16 boundary Casts; static-rule limitations retained.
• Quality: license, full Ruff, mypy 442, focused 39, Eval/recipes 704, Analyze 1529, models 1541, optim 876, and commands 3727 passed independently. The tester's raw 6 failed, 2536 passed, 9 skipped, 1 xfailed remains FAIL; I independently reproduced the exact same six failures on parent and candidate with identical mechanisms. The obsolete 1,429 total was not accepted.
• CI: all 9 exact-head checks completed SUCCESS, including Analyze (Python), lint, all five test jobs, CodeQL, and license/cla.
• Threads/comments: 0 review threads enumerated, 0 open; 0 line comments; 0 pre-existing conversation comments.
• feat(eval): add audio classification evaluation and music genre recipes #1388: still OPEN draft/unmerged at caefc9cc; all six shared production files are byte-identical. feat(eval): add audio classification evaluation and MMS language ID recipes #1389 is independently mergeable now, but if feat(eval): add audio classification evaluation and music genre recipes #1388 merges or main/head changes, route Planner rebase/dedupe and impacted validation before proceeding.

Coverage: full. Reachable verified tuples: CPU/cpu fp32 and CPU/cpu fp16. Deferred tuples: none.

No blocking findings. Keep the PR draft. Do not merge, ready, relabel, retire, or start a later model as part of this review.