recipe(wav2vec2): add Polish Wav2Vec2 ASR functional Eval - #1343
ssss141414 wants to merge 9 commits into
Conversation
|
Independent review of Draft #1343 at exact head P1 - Enforce the frozen selected-row identity contract before inference. An exact-head discriminator supplied distinct valid source indices but (separately) a missing dataset ID, missing audio path, duplicate audio path, and boolean dataset ID. Every case was accepted and each made two inference calls; for example, duplicate Required handoff:
Verified independently: exact stack and two-recipe range; recipe schema and FP16/FP32 semantics; all 81 sealed paths/bytes/SHA-256 values; pinned FLEURS source rows; exact corpus WER REQUEST_CHANGES |
78e0713 to
3d08948
Compare
|
Resolved the provenance defect reported in comment 5388804450 on the replacement dependency/model stack. Dependency fix on #1335: approved D6 is Fresh exact-D6 public evidence: focused CTC Model restack on #1343: exact C6 The pinned Polish two-row result remains WER Fresh exact-C6 public closure is The PR body and public commands now bind exact C6/D6/current-main identity and preserve the mixed historical execution provenance. Draft state and |
|
APPROVE
No actionable blocker remains. This is the single skill-level terminal reviewer opinion as a normal PR comment; it does not submit GitHub Review state, change Draft/readiness or labels, edit or resolve threads, push, or merge. |
Summary
This follow-up refreshes CPU FP32 and FP16 recipes for
jonatasgrosman/wav2vec2-large-xlsr-53-polish, a Polish Wav2Vec2 CTC automatic-speech-recognition checkpoint originally added in #1207. The model-owned contribution is Effort L0 / Outcome L0 and reaches Goal L3PASS_FUNCTIONAL_SMOKE: both CPU precisions pass build, performance, and PyTorch parity, while one representative FP32 CPU run proves end-to-end Polish ASR Eval operability. Exact C63d089489b96b913b49fcbe2210aa8bbeb2e43c3fis one commit directly atop approved D64b528133fef8cbeed4e9a3aba9be42d22a3c0221on currentmaine28b128f5c2f69ecb2d73b63d2aea0a5ee8bddd0; merge Draft #1335 first, then update this Draft so its upstream diff is exactly the two recipes.Model metadata
What the model does
This Apache-2.0 Wav2Vec2 XLSR-53 checkpoint transcribes Polish speech. It accepts a normalized 16 kHz waveform and emits frame-level logits over a 40-token Polish CTC vocabulary, which a CTC tokenizer collapses into text. This is verified from the pinned checkpoint/model card/config/tokenizer at
6b1cea36bd8bc5f65ec8081667cd9c0207d51970and the WinML inspection contract.Primary user stories
verified.mapped.Supported tasks
automatic-speech-recognitionWav2Vec2ForCTC; vendorWav2Vec2OnnxConfig; WinMLAutoModelForCTCinspectionverifiedModel architecture
Wav2Vec2ForCTCsource, and WinML inspection (verified; developer frame-logit storymapped).Validation and support evidence
1. Baseline
microsoft/winml-climain0876e5ae1c98a169a6137e092e0d7b30bf9cee33, WinML0.3.0.PASS; build complete in141.4 s(171.743236 swall),AutoModelForCTC, inputinput_valuesfloat32[1,16000], output CTC logits, 1,132/1,132 tagged export nodes, and a1,261,923,716-byte external-data sidecar. That baseline spent35.6 sin Optimize despite--no-optimize.automatic-speech-recognition, model classAutoModelForCTC, model typewav2vec2, opset 17, eager Transformers attention compatibility, fixed float32 waveform input[1,16000], and no Eval section.36.54 GB, below the planner's40 GBoptional-expansion target; no checkpoint baseline latency is claimed.mainhad no ASR evaluator. The initial dependency also selected optional-LM machinery withoutpyctcdecodeand rejected the valid ordinary-Wav2Vec2 tokenizer'starget_lang=null.automatic-speech-recognition; WinML added no new Optimum task (VENDOR-ONLY). Goal floor: L0.2. Goal
main; it required composite source-row identity, seeded selection, honest mixed provenance, targeted current-main VED quality, and replacement public authority.3. Outcome
PASS_FUNCTIONAL_SMOKE.cpu/cpu/fp32andcpu/cpu/fp16passed L0, L1, and L2. One representativecpu/cpu/fp32run passed L3. No tuples are deferred.e28b128f5c2f69ecb2d73b63d2aea0a5ee8bddd0-> approved D64b528133fef8cbeed4e9a3aba9be42d22a3c0221(8 commits) -> C63d089489b96b913b49fcbe2210aa8bbeb2e43c3f(1 commit). C6 parent is exactly D6; C6 tree is21ad6d9a44a2773ae033fcb5d82297a4168d3971; merge-base is currentmain.examples/recipes/jonatasgrosman_wav2vec2-large-xlsr-53-polish/cpu/cpu/automatic-speech-recognition_fp32_config.jsonandautomatic-speech-recognition_fp16_config.jsonin the same directory.gim-home/ModelKitArtifacts#254remains Open/Draft atb7efe2b77c59eafbd13c48c4cb9beb9336df8305, labeledmodel-scale-by-skill, mergeable, withGitOps/AdvancedSecurityterminalSUCCESS. It retainswav2vec2-028,wav2vec2-027, and_meta-112. Final C6 review found no new methodology friction, so no content change or duplicate finding was warranted.4. Per-EP/device/precision results and Functional smoke Eval
The full required CPU tuple matrix passed. FP16 reduced recorded RSS delta but was slower than FP32 on this CPU; no speedup is claimed.
input_valuesFLOAT[1,16000];logitsFLOAT[1,49,40]; 424 FLOAT + 11 INT64 initializerslogits[1,49,40]; 424 FLOAT16 + 11 INT64 initializers0.9999999999982816; max abs0.00007200241088867188; frame argmax agreement1.00.9999972943487657; max abs0.08947086334228516; frame argmax agreement1.0Historical provenance is explicit: L0 and L2 executed at
cac2526b620df08fdbcc26ef60fe590479e55a0e; L1 Perf executed at C5ac8acd690b48828eb7dfdd38c4aa5de9e3bcb14dusing artifacts fromcac2526b620df08fdbcc26ef60fe590479e55a0e; Analyze executed at C5ac8acd690b48828eb7dfdd38c4aa5de9e3bcb14dover the same retained artifacts. All retained evidence was rehashed successfully and is adopted for C6, never relabeled as fresh C6 execution. VED/export/models quality and public acquisition/identity are fresh on exact C6.Functional smoke Eval
CPU FP32 only; functional operability evidence only. This is not representative checkpoint accuracy, benchmark quality, or a claim that FP16 accuracy was measured. The retained artifact execution is from
cac2526b620df08fdbcc26ef60fe590479e55a0e; evaluator execution is from3440c5401d0111f06456b00662469daad2a6f0f6; the sealed result is adopted for exact C6 after D6 evaluator-series equality and full rehash. It is not fresh C6 model-stage execution.google/fleurs@70bb2e84b976b7e960aa89f1c648e09c59f894dd,pl_pl,validation, deterministic source order, no shuffle, non-streaming.audio-> normalized 16 kHz float32 waveform -> fixed[1,16000]windows -> logits[1,T,40];transcriptionreferences; argmax CTC collapse with blank/pad token 0; NFKC and whitespace normalization.0.7115384615384616; CER0.17794486215538846.10018492969996036091.wav10288018704489549018.wavAccounting closes exactly: requested
2, selected2, processed2, rejected0, skipped0, rejection reasons{}, predictions2, references2. Streaming and non-streaming probes selected the same ordered(source index, dataset ID, audio path)rows. D6 validates the composite identity before inference: missing/malformed index, dataset ID, or audio path and duplicate row/media identity fail with zero inference calls; repeated dataset IDs remain valid when source index and audio identity are distinct.5. Delta
The model-owned C6 delta is one commit, exactly two files,
42 insertions(+), 2 deletions(-), with stable patch ID4cb75500a8c98244c5e159ed93b3ba60c66b4575. It is patch-, subtree-, blob-, byte-, and semantic-equivalent to the accepted C5 recipe commit. There are no source, test, README, dependency, lock, workflow, or generated-report changes; the production recipe README is untouched./evalautomatic-speech-recognition;google/fleurs/pl_pl/validation; revision70bb2e84b976b7e960aa89f1c648e09c59f894dd; samples 2; shuffle false; seed 42; streaming false;audio-> input;transcription-> label/eval/quantnullfp16;fp16_keep_io_types=true; remaining generated quant fields retainedRecipe
loadervalues remainautomatic-speech-recognition/AutoModelForCTC/wav2vec2; input remains float32[1,16000]; output remainslogits; fp32quantremainsnull; fp16 retains FP32 I/O.Bug fix explanation for the stacked dependency
_selected_row_provenanceandcomputevalidated only integer source indices and duplicate indices, not the frozen composite(source index, dataset ID, audio path)identity.Validate CTC selected row identitiescommit validates every composite field before inference, rejects duplicate row/media identity, preserves repeated semantic IDs when source and audio identities differ, and emits the exact selected IDs, indices, and audio paths. The finalHonor seeded CTC evaluation shufflecommit closes deterministic seeded selection without changing that identity contract.50 passed, including malformed composite identity, duplicate audio identity, repeated dataset ID with distinct row/media identity, and pinned FLEURS identity tests; build compatibility116 passed; VED15 passed; five non-hardware CI partitions8,543 passed,78 skipped,3 xfailed,1 marker-deselected; all nine exact-head checks succeeded; independent review postedAPPROVEwith no blocker.6. Analyze summary — component level and op level
Static rule analysis completed for both retained built artifacts with no errors or warnings. These classifications describe static rule coverage, not runtime execution on accelerator EPs; CPUExecutionProvider is the only runtime exercised here. Analyze executed at C5
ac8acd690b48828eb7dfdd38c4aa5de9e3bcb14dover artifacts fromcac2526b620df08fdbcc26ef60fe590479e55a0eand was adopted for C6 after exact equivalence and rehash.Component-level summary
Op-level summary
CUDA/GPU, MIGraphX/GPU, TensorRT/GPU, DML/GPU, CPU/CPU, and VitisAI/NPU had no populated runtime-support classification in this rules set. Across 12 provider/device records per artifact there were zero analysis errors and zero warnings.
7. Reproduce commands
Fresh public acquisition and closure on exact C6 is
11/11 CLOSED: authority seals; public identity/parent/tree/merge-base/recipes; frozen initialization; self-contained public commands; clean candidate/dependency; authorized reuse rehash; prior r6 integrity rewalk; Polish L3 authority/provenance; exact read-only PR identity; terminal PR CI; successful PR CI. The fresh command rows were source binding, VED export, models partition, full Ruff, scoped Ruff format on C6 and D6, mypy, license, and both recipe config resolutions. The scoped format commands intentionally returned 1 for the same two inherited VED paths on D6 and C6; the sets are equal with no C6-only path, so this isPASS_NO_REGRESSION, not a clean format claim.Fresh exact-C6 results: VED export
15 passed; models partition1,538 passed,6 skipped,2 xfailed; Ruff check passed; mypy found no issues in440source files; license passed; both config resolutions passed and remained byte/semantic-equivalent to C5. All nine exact-head GitHub checks are terminalSUCCESS.