Add split Whisper encoder-decoder support

[ssss141414]

Summary

This contribution adds openai/whisper-medium automatic speech recognition as a split Whisper encoder and cached autoregressive decoder, with CPU FP32 and FP16 recipes and a generalized ASR evaluator. It ships at Effort/Outcome L2 with full required-tuple coverage and reaches the committed ceiling at L3 PASS. The final candidate builds and executes both precisions, preserves PyTorch parity through bounded generation, completes a real LibriSpeech functional smoke, and scores empty ASR hypotheses as deletion errors instead of aborting evaluation.

Model metadata

What the model does

Whisper Medium is a multilingual audio-to-text encoder-decoder Transformer. It converts up to 30 seconds of 16 kHz speech into log-Mel features, encodes them, and autoregressively generates text tokens for same-language transcription or speech translation.

• Evidence: pinned openai/whisper-medium revision abdf7c39ab9d0397620ccaea8974cc764cd0953e; the pinned model card describes an encoder-decoder model for ASR and speech translation trained on 680k hours of labelled speech; the pinned config has architectures=[WhisperForConditionalGeneration], is_encoder_decoder=true, and num_mel_bins=80.
• Confidence: verified.

Primary user stories

• A user supplies spoken audio to obtain a same-language text transcript for captions, indexing, or downstream text processing. Evidence: pinned model card transcription example and automatic-speech-recognition pipeline tag. Confidence: verified.
• A user supplies multilingual spoken audio to obtain an English translation for cross-language understanding. Evidence: the pinned model card describes multilingual speech translation controlled by decoder task tokens. Confidence: verified.

Supported tasks

Task	Support surfaces	Evidence	Confidence
automatic-speech-recognition	checkpoint, Transformers, Optimum ONNX, WinML	Checkpoint pipeline_tag=automatic-speech-recognition; Optimum TasksManager registers whisper for automatic-speech-recognition and automatic-speech-recognition-with-past; WinML main inspection resolved ASR through vendor defaults, but only to a generic monolithic inference class.	verified
speech-translation	checkpoint, Transformers	The pinned model card states that multilingual checkpoints perform speech translation using the decoder translate context token.	verified

Model architecture

A 769M-class sequence-to-sequence Transformer with a convolutional log-Mel encoder, 24 encoder layers, a cached autoregressive 24-layer decoder with self- and cross-attention, and a vocabulary projection head.

WhisperForConditionalGeneration
+-- WhisperModel
|   +-- Audio encoder
|   |   +-- Conv1d input projection (80 mel bins -> 1024)
|   |   +-- Strided Conv1d + positional embedding (1500 positions)
|   |   `-- Encoder layer x 24 (16-head self-attention; 1024 -> 4096 -> 1024 FFN)
|   `-- Autoregressive text decoder
|       +-- Token embedding (51865 x 1024) + positional embedding (448 positions)
|       `-- Decoder layer x 24 (16-head cached self-attention + cross-attention + 1024 -> 4096 -> 1024 FFN)
`-- Vocabulary projection (1024 -> 51865 logits)
    `-- Runtime generation policy: language/task/timestamp prompts, suppression, greedy/beam decoding

• Source/confidence: pinned checkpoint config at revision abdf7c39ab9d0397620ccaea8974cc764cd0953e, Transformers 4.57.6 modeling_whisper.py concrete class construction and forward paths, and the clean-main HTP hierarchy of 686 modules, 173 traced modules, and 2903 tagged ONNX nodes (verified).

Validation and support evidence

Baseline

The baseline is WinML 0.3.1 at current pinned main commit da5dbcd5812daaabb64b89b47c24db2da2d4e1c7. The refresh decision is PARTIAL-RERUN from last validated main 2ff69221a34700aaf3027dfc798a3511511b0835 and original baseline evidence commit f831830cbf2168b4833d44d8b3413ea0ecfd0dea. The complete 33-file moved-main range was classified by dependency reachability: model profile, Optimum probe, inspect/config, baseline FP32 build, baseline Eval schema, candidate FP32 build/parity, and candidate FP32 Analyze were retained with original provenance; baseline perf, FP16 build/parity/Analyze, all component perf, FP32 Eval, affected quality partitions, Ruff, mypy, and exact-SHA hosted checks were rerun.

Starting auto-configuration resolved automatic-speech-recognition through vendor defaults to a generic monolithic AutoModelForSpeechSeq2Seq/Whisper path. Optimum already exposed audio-classification, automatic-speech-recognition, automatic-speech-recognition-with-past, feature-extraction, and feature-extraction-with-past; WinML added no tasks, so the Optimum probe verdict was VENDOR-ONLY.

• Build floor, L0 PASS (reused from f831830c): the monolithic FP32 build completed in 309.5s. The missing artifact was reconstructed operationally on the exact base and matched the historical model content; this did not relabel the historical execution.
• Perf floor, L1 PASS (fresh under current-main code): generic monolithic forward mean 3077.163 ms, P50 3076.132 ms, throughput 0.32 samples/s, RSS total delta 372.34 MiB. This is not autoregressive transcription latency. GPU memory was unavailable under the current nullable/provenance schema.
• Eval floor (reused from 2ff69221): UNSUPPORTED-TASK, exit 2; automatic-speech-recognition was absent from the WinML Eval registry, and no moved-main change reached that registry or evaluator path.

Goal

• Committed Effort: L2.
• Committed Goal ceiling: L3.
• Committed Outcome: L2.
• Success definition: split Whisper encoder/decoder builds at CPU FP32 and FP16, executes with named-input numerical parity and bounded autoregressive generation, and runs one real LibriSpeech ASR functional smoke with WER and exact accounting.
• Ceiling change: none; planner revision 3 preserved E=L2, G=L3, O=L2.

Outcome

• Shipped tier: L2.
• Highest Goal verdict: L3 PASS.
• Coverage: full for required CPUExecutionProvider / cpu / fp32 and CPUExecutionProvider / cpu / fp16 tuples.
• Deferred tuples: none.
• Handoff: READY; no product blockers.
• Final provenance: candidate e31829c410abe7143b7798b8811c79efca093757 is based on da5dbcd5812daaabb64b89b47c24db2da2d4e1c7. The four contribution commits are PATCH-EQUIVALENT to pre-rebase head 922eb4489581e4e93bc3d972baaf76ddbfe178f9: every ordered range-diff row is =, stable patch IDs match, the 16-path name-status inventory matches, all 16 owned blobs match, and the integrated candidate tree equals the independently predicted merge tree. There were no conflicts, manual resolutions, dropped or added commits, or semantic patch changes.
• Shipped recipes: examples/recipes/openai_whisper-medium/cpu/cpu/automatic-speech-recognition_fp32_encoder_config.json, automatic-speech-recognition_fp32_decoder_config.json, automatic-speech-recognition_fp16_encoder_config.json, and automatic-speech-recognition_fp16_decoder_config.json in the same directory.
• Shipped code/tests: src/winml/modelkit/loader/task.py; src/winml/modelkit/models/hf/__init__.py; src/winml/modelkit/models/hf/whisper.py; src/winml/modelkit/models/winml/encoder_decoder.py; src/winml/modelkit/commands/build.py; src/winml/modelkit/eval/__init__.py; src/winml/modelkit/eval/evaluate.py; src/winml/modelkit/eval/automatic_speech_recognition_evaluator.py; src/winml/modelkit/utils/eval_utils.py; tests/unit/models/whisper/test_onnx_config.py; tests/unit/commands/test_build.py; tests/unit/eval/test_automatic_speech_recognition_evaluator.py.
• Knowledge capture: model findings whisper-001 through whisper-006, the whisper-002 counterexample update, methodology findings _meta-115 through _meta-117, and paired tester/reviewer contract updates are on separate draft ModelKitArtifacts PR #350, preserving the Lane A/Lane B boundary.

Per-EP/device/precision results and Functional smoke Eval

Goal ladder

Tier	Verdict	Evidence
L0	PASS	CPU FP32 split artifacts were reused from 1d2a2eac6516401570c4912c7baa8bc9ba50758a only after exact patch/tree/blob equivalence. CPU FP16 encoder and decoder were rebuilt on final SHA e31829c410abe7143b7798b8811c79efca093757; ONNX checker, public I/O, 48-in/48-out decoder cache, initializer precision, and adjacent external-data contracts passed.
L1	PASS	Baseline plus all four candidate component perf rows were freshly rerun with current latency, throughput, and nullable memory provenance.
L2	PASS	FP32 parity was reused from 1d2a2eac after equivalence. FP16 encoder, first decoder step, cached decoder step, and bounded 32-step generation parity were freshly rerun and passed on e31829c4.
L3	PASS	Final-SHA FP32 CPU LibriSpeech functional smoke produced WER 0.09090909090909091 with requested/processed/skipped 1/1/0. The formal mixed empty-hypothesis regression produced WER 2/3 with 2/2/0 accounting and no exception.

Per-tuple structure and perf

These are component/first-step measurements, not full autoregressive transcription latency. GPU memory is unavailable for these CPU runs.

Tier	EP / Device	Precision	Component	Verdict	Mean	P50	Throughput	RSS total delta
L0	CPUExecutionProvider / cpu	fp32	encoder	PASS	-	-	-	-
L0	CPUExecutionProvider / cpu	fp32	decoder	PASS	-	-	-	-
L0	CPUExecutionProvider / cpu	fp16	encoder	PASS	-	-	-	-
L0	CPUExecutionProvider / cpu	fp16	decoder	PASS	-	-	-	-
L1	CPUExecutionProvider / cpu	fp32	encoder	PASS	2468.873 ms	2523.547 ms	0.41 samples/s	364.75 MiB
L1	CPUExecutionProvider / cpu	fp32	decoder	PASS	405.558 ms	403.357 ms	2.47 samples/s	409.84 MiB
L1	CPUExecutionProvider / cpu	fp16	encoder	PASS	2665.425 ms	2600.702 ms	0.38 samples/s	833.93 MiB
L1	CPUExecutionProvider / cpu	fp16	decoder	PASS	355.741 ms	356.489 ms	2.81 samples/s	732.69 MiB

FP16 structural evidence: encoder 729 nodes, 368 FLOAT16 initializers, 614432768 external-data bytes, ratio 0.5000; decoder 1517 nodes, 584 FLOAT16 initializers, 1019509760 external-data bytes, ratio 0.5000, with 48 cache inputs and 48 outputs. Both adjacent external-data checks passed.

FP16 parity: encoder cosine 0.9999933185733331; first/cached logits cosine 0.9999999999696552 / 0.9999994068226346; minimum generation cosine 0.9999994068226346; worst max-absolute difference 0.03837871551513672; exact tokens and transcript matched across 32 steps.

Functional smoke Eval

PASS on final candidate e31829c410abe7143b7798b8811c79efca093757: FP32, CPUExecutionProvider, device cpu. This is functional-smoke operability evidence only, not representative accuracy or benchmark quality. No Eval accuracy was measured for FP16 or another EP/device tuple.

• Dataset: openslr/librispeech_asr revision 71cacbfb7e2354c4226d01e70d77d5fca3d04ba1, config clean, split validation, deterministic first row.
• Accounting: sample limit 1; requested 1; processed 1; skipped 0; selection seed 42.
• Fan-out caps: English transcribe prompt; timestamps disabled; 1 beam; one utterance capped at 30 seconds / 3000 log-Mel frames; generation length 32; return_sequences=1.
• Semantics: schema, label semantics, and prediction semantics verified; ASR mode seq2seq.
• Raw task metric: wer = 0.09090909090909091.
• Former blocker and new capability: baseline Eval returned UNSUPPORTED-TASK at exit 2; this contribution registers generalized CTC/seq2seq ASR evaluation with raw bytes/path audio decoding, mono conversion/resampling, Whisper feature extraction, bounded generation, transcript decoding, corpus WER, and exact accounting.
• Empty-hypothesis regression: references ['hello world', 'recognized'], hypotheses ['', 'recognized']; two word errors over three reference words; WER 2/3; requested/processed/skipped 2/2/0; no abort. The empty hypothesis contributes two deletions. Empty references and zero selected rows still fail closed.

Quality and hosted checks

• Ruff: PASS. Mypy: PASS over 446 source files.
• Focused ASR: 10 passed; Eval partition: 697 passed; optim partition: 883 passed; models partition: 1568 passed; analyze partition: 1529 passed.
• Commands local: 3968 passed, 1 failed; exact base reproduced the same installed OpenVINO plugin dependency failure. Remaining local: 982 passed, 1 failed; exact base reproduced the same PyPI TLS handshake failure. These are host/package conditions; the corresponding exact-SHA hosted test (commands) and test (remaining) checks succeeded.
• All nine checks attached to e31829c410abe7143b7798b8811c79efca093757 reached terminal SUCCESS: CodeQL, license/cla, lint, test (optim), test (commands), test (models), test (analyze), test (remaining), and Analyze (Python).

Delta

The monolithic baseline recipe is intentionally changed into separate encoder and decoder recipes. FP16 uses the same architecture-driven split contracts and additionally declares quant.mode=fp16. The production recipe README remains untouched.

Recipe	JSON pointer	Baseline value	Shipped value
FP32 encoder	/export/input_tensors	input_features float32 [1,80,3000] range [-1,1]; decoder_input_ids int32 [1,16] range [0,2]	input_features float32 [1,80,3000] range [0,1]
FP32 encoder	/export/output_tensors	logits; encoder_last_hidden_state	encoder_hidden_states
FP32 encoder	/loader	task=automatic-speech-recognition, model_class=AutoModelForSpeechSeq2Seq, model_type=whisper	task=feature-extraction, model_class=WhisperEncoderWrapper, model_type=whisper
FP32 decoder	/export/input_tensors	input_features float32 [1,80,3000]; decoder_input_ids int32 [1,16]	decoder_input_ids int32 [1,1]; encoder_hidden_states float32 [1,1500,1024]; decoder_attention_mask bool [1,448]; cache_position int64 [1]; past_{0..23}_{key,value} float32 [1,16,448,64]
FP32 decoder	/export/output_tensors	logits; encoder_last_hidden_state	logits; present_{0..23}_{key,value}
FP32 decoder	/loader	task=automatic-speech-recognition, model_class=AutoModelForSpeechSeq2Seq, model_type=whisper	task=text2text-generation, model_class=WhisperDecoderWrapper, model_type=whisper
FP16 encoder	/quant	null	mode=fp16, task=feature-extraction, model_id=openai/whisper-medium, model_type=whisper, fp16_keep_io_types=true
FP16 decoder	/quant	null	mode=fp16, task=text2text-generation, model_id=openai/whisper-medium, model_type=whisper, fp16_keep_io_types=true
FP16 encoder	/export and /loader	monolithic baseline export/loader	same split encoder values enumerated for FP32
FP16 decoder	/export and /loader	monolithic baseline export/loader	same split decoder values enumerated for FP32

The change remains reducibility-consistent with the charter. Recipe-free architecture acceptance passed and emitted split encoder/decoder artifacts; this is architecture acceptance, not a Goal-tier or precision verdict.

Bug fix explanation: architecture-specific split export

• (a) Symptom/minimal trigger: configuring or building openai/whisper-medium for ASR selected a generic monolithic forward graph rather than an end-to-end transcription-capable split composite.
• (b) Root cause: WinML had no Whisper architecture registration, component wrappers, or component I/O contracts, so vendor task resolution fell through to AutoModelForSpeechSeq2Seq.
• (c) Changed symbols/mechanism: WhisperEncoderWrapper, WhisperDecoderWrapper, WhisperEncoderIOConfig, WhisperDecoderIOConfig, and WinMLWhisperModel, with loader/HF registration, derive separate encoder and decoder exports and register the composite runtime.
• (d) Generality: dispatch is data-driven by Whisper architecture/task metadata; there is no checkpoint-ID branch.
• (e) Compatibility/blast radius: non-Whisper resolution remains registry-driven; registered Whisper ASR intentionally resolves to split export/runtime.
• (f) Regression evidence: L0 passes for CPU FP32/FP16 components, recipe-free acceptance passes, model/export suites pass, and all exact-SHA hosted checks succeed.

Bug fix explanation: shared encoder-decoder generation runtime

• (a) Symptom/minimal trigger: token-at-a-time Whisper generation requires audio encoder routing, prompt prefill, generation metadata, and 24 fixed self-attention KV pairs; the prior generic path lacked that complete contract.
• (b) Root cause: WinMLEncoderDecoderModel did not generalize all routing, prompt, and static-cache behavior required by Whisper.
• (c) Changed symbols/mechanism: WinMLEncoderDecoderModel generalizes encoder input routing, generation metadata, prompt prefill, and static-cache handling.
• (d) Generality: behavior is selected from component I/O and generation metadata, not checkpoint ID.
• (e) Compatibility/blast radius: Marian translation and vision encoder-decoder behavior is preserved; Whisper split generation is the intentional addition.
• (f) Regression evidence: FP32 and FP16 pass named-input encoder, first-step decoder, cached-step decoder, and 32-step generation parity with exact token/transcript matches; the fresh FP16 run completed in 28.540037900034804s.

Bug fix explanation: automatic speech recognition evaluator

• (a) Symptom/minimal trigger: winml eval --schema --task automatic-speech-recognition returned UNSUPPORTED-TASK with exit 2.
• (b) Root cause: ASR was absent from the evaluator registry, with no generalized path joining audio decoding, preprocessing, CTC/seq2seq inference, transcript decoding, WER, and accounting.
• (c) Changed symbols/mechanism: WinMLAutomaticSpeechRecognitionEvaluator, _asr_mode, _word_error_counts, _word_error_rate, _EVALUATOR_REGISTRY, and _DEFAULT_DATASETS add pinned LibriSpeech selection, bytes/path decoding, mono/resample/cap preprocessing, explicit CTC versus seq2seq dispatch, bounded generation, and corpus WER.
• (d) Generality: evaluator dispatch is architecture/output-contract driven rather than checkpoint hardcoding.
• (e) Compatibility/blast radius: existing CTC ASR retains frame-logit decoding; ASR task support and split seq2seq Whisper evaluation are intentional additions.
• (f) Regression evidence: real-data smoke exits 0, verifies semantics, processes 1/1/0, and emits WER 0.09090909090909091; focused ASR reports 10 passed, Eval reports 697 passed, and hosted checks succeed.

Bug fix explanation: empty ASR hypotheses

• (a) Symptom/minimal trigger: a decoded empty hypothesis paired with a non-empty reference raised an exception, aborting the dataset run instead of producing a WER result.
• (b) Root cause: compute rejected empty prediction strings before predictions/references reached the existing corpus word-error helpers, even though those helpers correctly represent the reference words as deletions.
• (c) Changed symbols/mechanism: WinMLAutomaticSpeechRecognitionEvaluator.compute removes only the empty-hypothesis exception; the mixed regression in test_automatic_speech_recognition_evaluator.py preserves empty predictions for corpus scoring.
• (d) Generality: the rule applies to any valid CTC or seq2seq ASR output contract and does not branch on checkpoint ID.
• (e) Compatibility/blast radius: non-empty ASR scoring is unchanged. Empty references and zero usable rows still fail closed; only empty decoded hypotheses become valid processed predictions.
• (f) Regression evidence: references ['hello world', 'recognized'] and hypotheses ['', 'recognized'] produce two errors over three reference words, WER 2/3, accounting 2/2/0, and no exception; focused ASR 10 passed, Eval 697 passed, Ruff/mypy pass, and all nine exact-SHA hosted checks succeed.

Bug fix explanation: --no-optimize stage control

• (a) Symptom/minimal trigger: the baseline build accepted --no-optimize but still executed the Optimize stage.
• (b) Root cause: the accepted stage-control value did not reach the actual optimization-stage sink in both Rich build pipelines.
• (c) Changed symbols/mechanism: _build_hf_pipeline and _build_onnx_pipeline thread --no-optimize / config.skip_optimize to the executing sink.
• (d) Generality: the fix applies to HF and ONNX stage control independent of model/task/precision.
• (e) Compatibility/blast radius: optimization remains unchanged when enabled; documented suppression now skips the real stage independently of FP16 conversion.
• (f) Regression evidence: commands/optim tests and their exact-SHA hosted partitions succeed; the local commands dependency failure reproduces on exact base and is unrelated.

Bug fix explanation: evaluator override typing repair

• (a) Symptom/minimal trigger: lint reported an incompatible prepare_pipeline override return type.
• (b) Root cause: the direct-inference evaluator intentionally has no pipeline, but its annotation did not match the established base-compatible pattern.
• (c) Changed symbols/mechanism: the override imports Pipeline under TYPE_CHECKING, uses Pipeline | None with the established suppression, and explicitly returns None.
• (d) Generality: the repair follows the evaluator abstraction and contains no checkpoint-specific behavior.
• (e) Compatibility/blast radius: runtime-neutral; no recipe, graph, inference, generation, decoding, or metric behavior changes.
• (f) Regression evidence: mypy passes over 446 source files, Ruff passes, focused ASR reports 10 passed, and all nine exact-SHA hosted checks succeed.

Analyze summary - component level and op level

Static Analyze status is PASS for all four artifacts; this is rule-based compatibility analysis, not runtime execution. FP32 Analyze was reused from c07a51acb6eaf6e19a9e0c50a0c6e80066a6e936 after exact equivalence, while FP16 encoder/decoder Analyze was freshly rerun on final candidate e31829c410abe7143b7798b8811c79efca093757. There are no component mapping gaps.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32 encoder	model.encoder.conv_frontend; model.encoder.layers[]	727 mapped, 0 unmapped	none
fp32 decoder	model.decoder.embeddings; model.decoder.layers[]; proj_out	1418 mapped, 0 unmapped	NvTensorRTRTX/GPU and OpenVINO/GPU: ScatterND, Tile, Where; QNN/GPU: Concat, Gather, LessOrEqual, ScatterND, Tile, Where
fp16 encoder	model.encoder.conv_frontend; model.encoder.layers[]	729 mapped, 0 unmapped	none
fp16 decoder	model.decoder.embeddings; model.decoder.layers[]; proj_out	1517 mapped, 0 unmapped	NvTensorRTRTX/GPU and OpenVINO/GPU: ScatterND, Tile, Where; QNN/GPU: Concat, Gather, LessOrEqual, ScatterND, Tile, Where

Op-level summary

Artifact	Graph	Dominant operators	Actionable EP findings
fp32 encoder	727 operators / 10 types	Reshape 264; Gemm 120; Transpose 97; MatMul 72; Add 49; LayerNormalization 49	none
fp32 decoder	1418 operators / 21 types	Reshape 484; Transpose 216; Gemm 192; MatMul 145; Add 97; LayerNormalization 73	NvTensorRTRTX/GPU and OpenVINO/GPU: ScatterND, Tile, Where; QNN/GPU: Concat, Gather, LessOrEqual, ScatterND, Tile, Where
fp16 encoder	729 operators / 11 types	Reshape 264; Gemm 120; Transpose 97; MatMul 72; Add 49; LayerNormalization 49	none
fp16 decoder	1517 operators / 21 types	Reshape 484; Transpose 216; Gemm 192; MatMul 145; Cast 100; Add 97	NvTensorRTRTX/GPU and OpenVINO/GPU: ScatterND, Tile, Where; QNN/GPU: Concat, Gather, LessOrEqual, ScatterND, Tile, Where

Rule-less CUDAExecutionProvider/GPU, MIGraphXExecutionProvider/GPU, TensorrtExecutionProvider/GPU, and DmlExecutionProvider/GPU rows have no runtime-support classification. The actionable decoder findings above preserve partial, unsupported, and unknown static-rule outcomes; they are not runtime-support claims.

Reproduce commands

$OUT='temp/whisper-medium-support-repro'
uv run winml build -c examples/recipes/openai_whisper-medium/cpu/cpu/automatic-speech-recognition_fp32_encoder_config.json -m openai/whisper-medium -o $OUT/fp32/encoder
uv run winml build -c examples/recipes/openai_whisper-medium/cpu/cpu/automatic-speech-recognition_fp32_decoder_config.json -m openai/whisper-medium -o $OUT/fp32/decoder
uv run winml build -c examples/recipes/openai_whisper-medium/cpu/cpu/automatic-speech-recognition_fp16_encoder_config.json -m openai/whisper-medium -o $OUT/fp16/encoder --precision fp16
uv run winml build -c examples/recipes/openai_whisper-medium/cpu/cpu/automatic-speech-recognition_fp16_decoder_config.json -m openai/whisper-medium -o $OUT/fp16/decoder --precision fp16
uv run winml perf -m $OUT/fp16/encoder/model.onnx --ep cpu --device cpu --iterations 3 --warmup 1 --output $OUT/perf-fp16-encoder.json --overwrite
uv run winml eval -m encoder=$OUT/fp32/encoder/model.onnx -m decoder=$OUT/fp32/decoder/model.onnx --model-id openai/whisper-medium --task automatic-speech-recognition --device cpu --ep cpu --dataset openslr/librispeech_asr --dataset-name clean --dataset-revision 71cacbfb7e2354c4226d01e70d77d5fca3d04ba1 --split validation --samples 1 --no-shuffle --column max_audio_seconds=30 --column max_new_tokens=32 --column language=english --column generation_task=transcribe --output $OUT/eval.json --overwrite

[ssss141414]

Independent reviewer verdict: REQUEST_CHANGES

Reviewed exact candidate SHA: c07a51acb6eaf6e19a9e0c50a0c6e80066a6e936

Blocking issue

• Planner — baseline-currentness gate: the charter records baseline.main_commit=f831830cbf2168b4833d44d8b3413ea0ecfd0dea with a FRESH baseline at that SHA, but fetched origin/main is now 2ff69221a34700aaf3027dfc798a3511511b0835. The 17-file moved-main range directly changes this PR's evaluator/workflow dependency surface, including .github/workflows/modelkit-ci.yml, src/winml/modelkit/commands/eval.py, src/winml/modelkit/eval/config.py, src/winml/modelkit/eval/evaluate.py, and associated tests. The charter has no complete impact attestation for f831830c..2ff69221. Please issue a replacement charter with a fresh current-main baseline, or provide the required file-by-file moved-main impact attestation, candidate-patch-equivalence result, and honest reused/rerun provenance; then refresh all downstream evidence invalidated by that decision.

Verified passing evidence

• Live PR remains draft, targets main, carries model-scale-by-skill, and remains at the reviewed SHA.
• Conversation enumeration: 0 issue comments, 0 line comments, 0 review threads, 0 open threads.
• All 9 exact-SHA GitHub checks are COMPLETED/SUCCESS.
• Ruff passed; mypy passed for 445 source files.
• Affected partitions: models 1568 passed, 7 skipped, 1 xfailed; commands 3824 passed, 9 skipped; remaining 982 passed, 2 skipped, 1 deselected. The one local failure in each latter partition was reproduced unchanged on clean current main (broken host OpenVINO package and PyPI TLS respectively), so neither is a PR regression.
• Coverage is full for required CPU FP32 and FP16 tuples with no deferred tuples: L0, L1, and L2 pass for both; the representative final-SHA FP32 CPU L3 smoke passes.
• I independently ran ASR schema/config commands, inspected all source and recipe changes, verified sealed hashes and four retained ONNX graphs, checked named I/O and precision, and confirmed the architecture-registered split design and no-optimize propagation are sound.
• Lane A methodology provenance is live at gim-home/ModelKitArtifacts#350, head 9177d45d9ec2bc79e46be5a990dc631c8304b1d1, with the two declared commits.

This is a skill-level reviewer opinion posted as a normal PR comment, not a GitHub Review state. No product code or PR metadata was modified.

[ssss141414]

Explainer resolution of baseline-currentness request

Resolved the provenance gap raised in the prior REQUEST_CHANGES comment.

• Planner revision 2 selected PARTIAL-RERUN, classified the complete 17-file moved-main range, retained original evidence provenance at f831830cbf2168b4833d44d8b3413ea0ecfd0dea, and reran the invalidated baseline Eval stage against current main 2ff69221a34700aaf3027dfc798a3511511b0835.
• Producer revision 3 rebased the three contribution commits onto 2ff69221 and proved candidate ce286318bbdec91edfff77d68b3de7c4519ef950 PATCH-EQUIVALENT to old head c07a51acb6eaf6e19a9e0c50a0c6e80066a6e936, with no conflicts, manual resolutions, added commits, or semantic patch changes.
• Tester revision 4 reran L3, affected tests/static checks, the commands partition, and all nine exact-SHA hosted gates; L0-L3 remain PASS, coverage remains full, and blockers are empty.
• The PR body now carries the current/original baseline provenance, reuse/rerun split, rebase proof, current candidate SHA, and refreshed validation results while preserving the existing model, tuple, delta, Analyze, command, and Lane A evidence.

This is an explainer provenance update posted as a normal PR comment. The prior reviewer opinion remains the latest reviewer verdict pending independent re-review; no GitHub Review state was created or changed.

[ssss141414]

Independent reviewer verdict: APPROVE

APPROVE for microsoft/winml-cli#1407 at exact head ce286318bbdec91edfff77d68b3de7c4519ef950 against main 2ff69221a34700aaf3027dfc798a3511511b0835.

• Live shipment: OPEN draft, base main, model-scale-by-skill label present, and the live body exactly matches explainer revision 2.
• Conversation: 2 issue comments enumerated (the prior reviewer REQUEST_CHANGES and the explainer's resolution), 0 line comments, 0 GitHub Reviews, 0 review threads, and 0 unresolved threads. GraphQL reported hasNextPage=false.
• Hosted checks: all 9 exact-SHA checks are COMPLETED/SUCCESS: Analyze (Python), lint, test (analyze), test (models), test (optim), test (commands), test (remaining), CodeQL, and license/cla.
• Coverage: full for CPUExecutionProvider/cpu/fp32 and CPUExecutionProvider/cpu/fp16; L0, L1, and L2 pass for both tuples, and the representative final-SHA FP32 CPU L3 functional smoke passes. Deferred tuples: none.
• Moved-main gate: planner revision 2's PARTIAL-RERUN is correct. The complete 17-file f831830c..2ff69221 range was classified; affected Eval/workflow stages reran, unaffected build/perf/L2/Analyze evidence retained original provenance, and the rebased contribution is patch-equivalent (range-diff all =, 16/16 owned blobs match, expected tree equals candidate tree).
• Independent local validation from a fresh detached worktree: Ruff passed; mypy passed in 445 source files; focused ASR/shared-Eval tests passed 234; models partition passed 1568 with 7 skips and 1 xfail. Commands reached 3835 passes with one OpenVINO DLL host failure reproduced unchanged on exact base; exact-SHA hosted test (commands) passed.
• Engineering review: the 16-file diff is data-driven by Whisper architecture/task metadata, has no checkpoint-ID routing or production README change, preserves the shared encoder-decoder runtime body while separating generation policy, bounds audio/generation, keeps CTC and seq2seq decoding explicit, and covers both stage-disable sinks and sibling/shared partitions.
• Artifact/evidence review: all supplied manifests and provenance hashes verify; all four ONNX graphs load with expected IR/opset/named I/O, fp16 initializers and half-sized external data are confirmed, component/op Analyze and HTP/build-config evidence is complete, and L2 exact token/transcript parity validates the recipe BOOL-to-graph FLOAT mask lowering.
• Lane A: gim-home/ModelKitArtifacts#350 is live, draft, based on main, labeled model-scale-by-skill, at 9177d45d9ec2bc79e46be5a990dc631c8304b1d1; _meta-115, _meta-116, and whisper-001 through whisper-005 are present with the cited producer/reviewer contract updates.

This is a skill-level opinion in a normal PR conversation comment. It does not change GitHub Review state, draft/readiness, labels, body, branch, or thread resolution.

[Qiong Wu (qiowu) (DingmaomaoBJTU)]

Reviewed the split Whisper runtime/export integration, ASR evaluator, and build-stage forwarding. The reported CI checks are green, but I am withholding approval for the evaluator behavior below. A minimal execution of the actual compute method and WER helpers reproduces the issue; full Whisper model export/inference was not rerun locally.

[ssss141414]

Independent reviewer verdict: APPROVE

APPROVE for microsoft/winml-cli#1407 at exact head e31829c410abe7143b7798b8811c79efca093757 against main da5dbcd5812daaabb64b89b47c24db2da2d4e1c7.

• Live shipment: OPEN draft, base main, model-scale-by-skill present, expected head, and live body exactly matches explainer revision 3.
• Conversation gate: complete pagination enumerated 3 GitHub Reviews, 3 issue comments, 3 inline comments, and 1 review thread. The prior empty-hypothesis thread is resolved/outdated; unresolved threads: 0.
• Hosted gate: all 9 checks attached to the exact head are COMPLETED/SUCCESS: Analyze (Python), lint, test (analyze), test (models), test (optim), test (commands), test (remaining), CodeQL, and license/cla.
• Handoff integrity: planner r3, producer r4, tester r5, learner r4, and explainer r3 manifests all verify: 136 sealed entries, no hash mismatch, no missing file, and no unsealed file.
• Moved-main gate: PARTIAL-RERUN is correct. All 33 moved-main files are classified with no UNKNOWN; perf and FP16 quantization-dependent build/parity/Analyze were rerun with downstream quality. Reused profile/config, baseline L0, and FP32 build/parity/Analyze retain explicit historical execution provenance.
• Patch equivalence: four ordered range-diff rows are =, all stable patch IDs match, all 16 owned blobs match, and expected tree b0448103b330588573f928ac90f1af17f0268efa equals the candidate tree. Older SHAs in the body are explicitly historical provenance; the final thread reply contains no stale SHA claim.
• Engineering review: the architecture-specific split Whisper export and generation policy are correctly isolated over the shared encoder-decoder core. ASR raw-audio decoding, bounded deterministic generation, corpus WER, and exact accounting are sound. Empty hypotheses score as deletions; empty references and zero selected rows fail closed. No checkpoint-ID routing, production README edit, or unrelated scope was found.
• Independent local gates: Ruff passed; mypy passed over 446 source files; focused ASR/Whisper tests passed 16. Exact models, analyze, and optim partitions passed. commands reached 3968 passes with one missing-OpenVINO-DLL host failure reproduced on exact base. Exact remaining reached 982 passes with one PyPI TLS failure reproduced on exact base. Hosted exact-SHA commands/remaining jobs passed.
• Coverage: full for CPUExecutionProvider/cpu/fp32 and CPUExecutionProvider/cpu/fp16; L0-L3 PASS; deferred tuples: none; blockers: none.
• Artifacts and model evidence: the independent ONNX inspector passed. FP16 decoder has 584 FLOAT16 initializers, 48 cache inputs/outputs, adjacent external data, and about half the FP32 external-data size. Fresh FP32 encoder/decoder means are 2468.873 / 405.558 ms; FP16 means are 2665.425 / 355.741 ms. FP16 parity covers 32 generation steps with exact tokens/transcript.
• Functional smoke: pinned LibriSpeech clean/validation@71cacbfb7e2354c4226d01e70d77d5fca3d04ba1, one real remote audio row, WER 0.090909..., requested/processed/skipped 1/1/0, 30-second audio cap, 32-token cap, one beam. This proves operability, not representative accuracy. The mixed empty/non-empty regression independently confirms WER 2/3 and 2/2/0 accounting.
• Analyze: complete component mappings and op summaries are retained for encoder 729 ops / 11 types and decoder 1517 / 21, with seven requested EP classifications per artifact. Rules-unavailable rows remain static classifications and are not presented as runtime support.

This is a skill-level reviewer opinion posted as one normal PR issue comment. It does not submit GitHub Review state or mutate draft/readiness, labels, body, branch, merge state, or review threads.