feat(eval): add translation evaluation and refresh opus-mt-en-ru recipes

[ssss141414]

Summary

This adds generalized translation evaluation for WinML encoder-decoder models, refreshes the four existing Helsinki-NLP/opus-mt-en-ru CPU recipes, and fixes encoder cross-attention mask alignment during generation. The evaluator uses TorchMetrics corpus SacreBLEU with 13a tokenization and standard chrF2; no dependency is added. The contribution is Effort L2 / Outcome L2 and reaches L3 PASS with full planned CPU fp32/fp16 coverage; the two-row Eval is functional-smoke evidence only, not representative translation quality.

Model metadata

What the model does

Helsinki-NLP/opus-mt-en-ru accepts English text and autoregressively produces Russian text using a Marian encoder-decoder translation model. Evidence: pinned checkpoint Helsinki-NLP/opus-mt-en-ru at revision bb09c99d180016eac6819df3dae68edb1690fdee, whose Hub metadata identifies translation, English, Russian, and Apache-2.0; pinned config identifies MarianMTModel, model_type=marian, and is_encoder_decoder=true. Confidence: verified.

Primary user stories

• A user supplies English text to obtain a Russian translation for cross-language communication or downstream text processing. Evidence: checkpoint translation metadata and pinned model-card example inputs. Confidence: verified.

Supported tasks

Task	Support surface	Evidence	Confidence
translation	checkpoint, Transformers, WinML composite	Hub pipeline tag, AutoModelForSeq2SeqLM mapping, and WinML composite registry	verified
text2text-generation	Transformers, Optimum ONNX, WinML decoder	Optimum MarianOnnxConfig vendor registry and WinML MarianDecoderIOConfig override	verified
feature-extraction	Optimum ONNX, WinML encoder	Optimum MarianOnnxConfig vendor registry and WinML MarianEncoderIOConfig override	verified

Model architecture

MarianMTModel (vocab 62,518; hidden width 512)
|-- Encoder graph (batch 1, source length 512)
|   |-- Token + sinusoidal position embeddings
|   `-- Encoder layer x6
|       |-- Self-attention (8 heads)
|       `-- Feed-forward + residual + LayerNorm
`-- Cached decoder graph (batch 1, one token per step, cache 512)
    |-- Token + sinusoidal position embeddings
    |-- Decoder layer x6
    |   |-- Masked self-attention (8 heads, static KV cache)
    |   |-- Encoder cross-attention
    |   `-- Feed-forward + residual + LayerNorm
    `-- LM head -> logits [1, 1, 62,518]

• Source/confidence: pinned config dimensions, Transformers MarianMTModel source, and fresh HTP/ONNX artifacts (verified).

Validation and support evidence

1. Baseline

The baseline was fully rerun on main commit be3e59dd412d4918c5a852aa4d8f0207c34aaf6f with WinML 0.0.1.dev0; broad runtime, dependency, Marian, generation, Eval, and Analyze changes since the prior baseline made reuse unsafe. CPU fp32 auto-build passed in 137.19 s and emitted opset-17 encoder and cached-decoder artifacts at fixed batch 1, source/cache length 512, and vocabulary 62,518. Baseline perf passed: encoder average/p50 69.76/69.93 ms, 14.34 samples/s, RAM +64.4 MB; decoder average/p50 21.68/20.78 ms, 46.12 samples/s, RAM +21.2 MB. Encoder parity passed on a pinned real row with cosine 0.9999999999993413 and max absolute error 5.245208740234375e-06.

Baseline two-row generation mechanically ran but collapsed both inputs to the same low-quality short Russian output. Translation Eval was UNSUPPORTED-TASK before dataset loading; an ad hoc probe returned SacreBLEU-13a 0.0 and chrF2 0.012449 raw (1.2449366 on the 0-100 scale), as functional smoke only. Analyze produced parseable partial-success JSON: encoder 197 nodes / 15 operator types and decoder 378 nodes / 20 operator types. Optimum exposed feature-extraction/text-generation/text2text-generation with and without past; WinML added no keys but replaced vendor feature-extraction and text2text-generation configurations with Marian-specific I/O overrides (VENDOR+OVERRIDE).

2. Goal

• Effort: L2
• Goal ceiling: L3
• Outcome: L2
• Committed success definition: refresh the four existing CPU encoder/decoder fp32/fp16 recipes; add a generalized, explicit-direction translation evaluator with bounded static-batch generation and corpus SacreBLEU-13a/chrF2; preserve existing behavior; then attempt CPU fp32/fp16 L0-L2 and one final-candidate CPU fp32 two-row L3 functional smoke.
• Required tuples: CPUExecutionProvider / cpu / fp32 and fp16. No deferred tuples.

No downstream role silently re-tiered the frozen charter.

3. Outcome

L3 PASS, full planned coverage, no deferred tuples. The final candidate is 3f5f321e4721e3022f7f50514e8cfcfb9ad4570d (tree ddf4c7810f2029937798e3ca1c03e4e6595a07af) on base be3e59dd412d4918c5a852aa4d8f0207c34aaf6f; it is clean and unpushed in the sealed handoff.

Shipped product paths comprise the four existing Model25 CPU recipes plus generalized Eval/runtime code and focused tests. There is no hardcoded checkpoint-name branch, no dependency addition, no README change, and no duplicate recipe path. Learner findings marian-010 through marian-013 and methodology findings _meta-117 and _meta-118 were published separately in Lane A draft PR #314 at commit 2f4c04fa1ff3b427d8bb9c39f7c95c323d178ec3; no skill files are included in this model contribution.

Quality gates

• Focused evaluator/model/recipe/regression suite: 72 passed in 32.41 s.
• Ruff: All checks passed. License headers: Passed. Translation Eval schema dispatch: PASS.
• Mypy: Success: no issues found in 443 source files.
• Canonical partitions: 1529 passed, 45 skipped (Analyze); 1542 passed, 7 skipped, 1 xfailed (models); 876 passed, 16 skipped, 1 xfailed (optim); 3706 passed, 9 skipped, 2 warnings (commands/config/build/compiler/session/Eval); 964 passed, 2 skipped, 1 deselected, 1 warning (remaining unit/regression/CLI).

4. Per-EP/device/precision results and Functional smoke Eval

Goal ladder and artifact signatures

L0 and L1 are PRESERVED, not re-executed on the final runtime-repair SHA. They executed on 92387df1fbe50b628ac38574cdcb0fa992247088 and were reused only after 38 sealed artifact files and all recipes rehashed exactly and the repair was proven evaluator/runtime-only with no config, export, optimizer, recipe, lock, artifact-input, or component-perf impact. The execution SHA remains 92387df1...; it is not relabeled as final-candidate execution.

Tier	EP / device	Precision / component	Verdict	Exact artifact signature, size, and realized dtype
L0	CPUExecutionProvider / cpu	fp32 encoder	PASS (preserved)	Inputs input_ids[1,512], attention_mask[1,512]; output encoder_hidden_states; model 67,274 bytes + model.onnx.data 204,742,656 bytes; initializers FLOAT x102, INT64 x29; opset 17
L0	CPUExecutionProvider / cpu	fp32 decoder	PASS (preserved)	Inputs decoder_input_ids[1,1], encoder_hidden_states[1,512,512], attention_mask[1,512], decoder_attention_mask[1,512], cache_position[1], 12 KV tensors past_{0..5}_{key,value}[1,8,512,64]; outputs logits + 12 present_{0..5}_{key,value}; model 146,157 bytes + model.onnx.data 358,291,672 bytes; initializers FLOAT x166, INT64 x43; opset 17
L0	CPUExecutionProvider / cpu	fp16 encoder	PASS (preserved)	Inputs input_ids[1,512], attention_mask[1,512]; output encoder_hidden_states; model 102,432,639 bytes; initializers FLOAT16 x102, INT64 x29; opset 17
L0	CPUExecutionProvider / cpu	fp16 decoder	PASS (preserved)	Same decoder input/output signature as fp32; model 149,324 bytes + model.onnx.data 179,156,076 bytes; initializers FLOAT16 x166, INT64 x43; opset 17

All four artifact checks also passed co-located external-data, vocabulary 62,518, and realized-precision validation.

Perf

Tier	EP / device	Precision / component	Verdict	Mean	p50	Throughput	RAM total
L1	CPUExecutionProvider / cpu	fp32 encoder	PASS (preserved)	69.23 ms	68.24 ms	14.45 samples/s	64.4 MB
L1	CPUExecutionProvider / cpu	fp32 decoder	PASS (preserved)	22.51 ms	21.75 ms	44.42 samples/s	21.2 MB
L1	CPUExecutionProvider / cpu	fp16 encoder	PASS (preserved)	89.09 ms	88.50 ms	11.22 samples/s	74.8 MB
L1	CPUExecutionProvider / cpu	fp16 decoder	PASS (preserved)	24.92 ms	24.08 ms	40.13 samples/s	27.4 MB

L2 parity and generation

Precision	Encoder cosine / max abs	Decoder-logit cosine / max abs	Greedy token parity	Translation parity
fp32	0.9999999999993413 / 5.245208740234375e-06	0.9999999999986515 / 1.71661376953125e-05	exact	exact
fp16	0.999998659921451 / 0.007945060729980469	0.9999992557247197 / 0.014481067657470703	exact	exact

For both fp32 and fp16, ONNX and pinned PyTorch produced these exact, distinct full greedy sequences and decoded texts:

1. Tokens (32): [62517, 71, 2071, 575, 497, 221, 11, 20840, 30, 164, 3427, 2, 119, 26, 3712, 41, 2207, 806, 8432, 1300, 2, 119, 3983, 234, 38192, 53, 4410, 1046, 17820, 146, 3, 0]
   Text: "У нас есть 4-месячные мыши, которые не диабиотичны, которые раньше были диабетиками", добавил он.
2. Tokens (55): [62517, 1094, 5192, 1432, 21824, 261, 779, 2, 13172, 25978, 443, 374, 21647, 3085, 48, 1974, 10908, 6, 37750, 43669, 30, 2, 10383, 1561, 806, 6959, 847, 2, 7, 15382, 42, 4421, 10997, 7, 22526, 6876, 40306, 48, 7215, 38192, 41, 28840, 53, 2, 28, 2549, 145, 401, 4275, 6, 4790, 1029, 11078, 3, 0]
   Text: Доктор Эхуд Ур, профессор медицины Далхаузийского университета в Галифаксе, Новая Шотландия, и председатель клинического и научного отделения Канадской ассоциации диабета предупредили, что исследования все еще находятся в начале своего существования.

Functional smoke Eval

PASS on final candidate 3f5f321e4721e3022f7f50514e8cfcfb9ad4570d, FP32 CPU only. Before this contribution, WinML rejected translation as an unsupported Eval task. The generalized evaluator now performs explicit source/reference extraction, bounded composite generation, decoding, exact accounting, and corpus metrics through TorchMetrics SacreBLEUScore(tokenize='13a') and CHRFScore(n_char_order=6, n_word_order=0, beta=2.0) on a 0-100 scale.

• Dataset: gsarti/flores_101 source revision bc58ae43b22607b3e1e2bf3ae1bc5cb053495abb, immutable Parquet revision 1a45e707ea4b5ea3d6c71341f18bfca6a6e356a3, config all, split devtest, deterministic rows 1-2, sentence_eng -> sentence_rus.
• Caps: max source tokens 128; max new tokens 64; beams 1; return sequences 1; sampling false; two generated sequences; maximum 128 total decoder steps.
• Accounting: attempted 2, evaluated 2, skipped 0.
• Result: SacreBLEU-13a 9.7950; chrF2 56.8509. Independent recomputation returned the same values.

Row	Source	Reference	Prediction	Generated tokens
1	"We now have 4-month-old mice that are non-diabetic that used to be diabetic," he added.	"Теперь у нас есть четырёхмесячные мыши, у которых больше нет диабета", — добавил он.	"У нас есть 4-месячные мыши, которые не диабиотичны, которые раньше были диабетиками", добавил он.	32
2	Dr. Ehud Ur, professor of medicine at Dalhousie University in Halifax, Nova Scotia and chair of the clinical and scientific division of the Canadian Diabetes Association cautioned that the research is still in its early days.	Согласно предупреждению доктора Эхуда Ура (Ehud Ur), профессора медицины в Университете Делхаузи в Галифаксе (Новая Шотландия) и председателя клинико-научного отдела Канадской диабетической ассоциации, исследования все еще находятся на начальной стадии.	Доктор Эхуд Ур, профессор медицины Далхаузийского университета в Галифаксе, Новая Шотландия, и председатель клинического и научного отделения Канадской ассоциации диабета предупредили, что исследования все еще находятся в начале своего существования.	55

This proves end-to-end evaluator operability only. Two adjacent rows from one document are not representative sampling, and these values are not benchmark-quality or representative accuracy claims. Eval was not repeated for fp16 and no fp16 accuracy claim is made.

5. Delta

Commit delta

Commit	Purpose	Merge-safe disposition
caa7dfb972b0c8cc1c5c878defcde790139eaf32	Generalized bounded translation evaluator	Droppable if PR #1213 or equivalent generalized translation Eval merges first
92387df1fbe50b628ac38574cdcb0fa992247088	Four Model25 eager-attention recipe refreshes	Retain and rebase independently
3f5f321e4721e3022f7f50514e8cfcfb9ad4570d	Preserve encoder mask alignment during generation	Retain unless merged runtime already right-pads encoder attention_mask equivalently

Changed files

• examples/recipes/Helsinki-NLP_opus-mt-en-ru/cpu/cpu/translation_fp16_decoder_config.json
• examples/recipes/Helsinki-NLP_opus-mt-en-ru/cpu/cpu/translation_fp16_encoder_config.json
• examples/recipes/Helsinki-NLP_opus-mt-en-ru/cpu/cpu/translation_fp32_decoder_config.json
• examples/recipes/Helsinki-NLP_opus-mt-en-ru/cpu/cpu/translation_fp32_encoder_config.json
• src/winml/modelkit/eval/__init__.py
• src/winml/modelkit/eval/evaluate.py
• src/winml/modelkit/eval/metrics/translation.py
• src/winml/modelkit/eval/translation_evaluator.py
• src/winml/modelkit/models/winml/encoder_decoder.py
• src/winml/modelkit/utils/eval_utils.py
• tests/unit/eval/test_translation_evaluator.py
• tests/unit/models/winml/test_composite_from_pretrained.py

All four existing recipes change only /export/compatibility/transformers_attention from absent (null) to "eager", matching current auto-config; fp16 quantization is preserved. The evaluator registry is additive; translation direction comes from explicit source/reference fields, bounded capacities come from model metadata, metrics are corpus-aggregated using the existing TorchMetrics dependency, failures propagate, and zero evaluated rows fails closed. examples/recipes/README.md remains untouched.

Bug fix explanation

• Symptom and trigger: component parity passed, but full greedy generation for two distinct short, right-padded encoder inputs collapsed to the same incorrect token sequence/text in fp32 and fp16.
• Root cause: encoder inputs and hidden states were right-padded to the static 512-token width, while _run_decoder applied cache-oriented left-padding to every decoder feed. This moved each short encoder attention mask from active source positions 0..30 / 0..47 to 481..511 / 464..511, so decoder cross-attention ignored the actual source tokens. Already-512-wide one-step parity inputs hid the defect.
• Fix mechanism: WinMLEncoderDecoderModel now preserves right-padding for the encoder cross-attention mask while retaining left-padding where the static decoder cache requires it; the regression exercises test_encoder_decoder_preserves_encoder_alignment_across_independent_rows.
• Generalization: the rule follows tensor role and static shape/capacity metadata. It does not inspect checkpoint, organization, language, dataset, or recipe names.
• Compatibility/blast radius: export, optimizer, recipe, lock, artifact signatures, cache identity/progression, static batch-one fan-out, and existing evaluator behavior are preserved. The intentional changes are translation Eval support, the explicit recipe compatibility field, and correct mask alignment for short encoder sequences.
• Regression evidence: focused test passed; repaired masks remain at 0..30 / 0..47; two source rows produce distinct encoder states and distinct outputs; full bounded fp32 and fp16 ONNX token sequences and decoded translations exactly match pinned PyTorch.

PR #1213 overlap

As last checked, microsoft/winml-cli PR #1213 was OPEN, DRAFT, DIRTY/CONFLICTING at head a74eb377a2d43468c047ea10649aa52faec48991; no Model25 PR existed. It overlaps the generalized evaluator. If #1213 merges first, drop caa7dfb9, rebase the recipe/runtime commits onto current main, return to Planner for dependency-aware impact analysis, and rerun Eval plus every invalidated check. Do not drop 3f5f321e unless the merged encoder-decoder runtime proves equivalent encoder-mask alignment.

6. Analyze summary — component level and op level

Static rule analysis is ANALYZE-PARTIAL-SUCCESS: both all-EP commands emitted complete seven-target JSON, while exit 1 reflects providers without shipped rule data. This is compatibility analysis, not runtime execution.

Component-level summary

Artifact	Architecture coverage	Mapping	Actionable EP findings
fp32 encoder	token/position embeddings; encoder layers x6; self-attention and feed-forward regions	197 nodes / 15 op types mapped through frozen scope-name hierarchy	No actionable partial/unsupported type in shipped TensorRT RTX, QNN, or OpenVINO rules; CUDA, MIGraphX, TensorRT, and DML runtime providers were absent
fp32 cached decoder	token/position embeddings; decoder layers x6; self/cross-attention; static KV cache; LM head; runtime generation loop	378 nodes / 20 op types mapped through frozen scope-name hierarchy	QNN partial: Expand, Cast, Concat; ScatterND unknown for QNN, OpenVINO, and TensorRT RTX; CUDA, MIGraphX, TensorRT, and DML runtime providers were absent

Op-level summary

Artifact	Graph	Dominant operators	EP roll-up
fp32 encoder	197 operators / 15 types	Reshape 61; Gemm 36; Transpose 24; Add 19; Mul 13; MatMul 12; LayerNormalization 12	TensorRT RTX, QNN, and OpenVINO rule classifications complete; runtime presence was observed only where reported by the static analyzer and is not an execution claim
fp32 cached decoder	378 operators / 20 types	Reshape 112; Gemm 60; Transpose 54; Add 33; MatMul 25; Mul 20; LayerNormalization 18; ScatterND 13	QNN partial for Expand/Cast/Concat; ScatterND unknown for QNN/OpenVINO/TensorRT RTX; absent-rule/runtime providers remain unknown rather than unsupported

Provider absence and missing rule data are distinct from an unsupported verdict. No accelerator runtime support is claimed from static rules.

7. Reproduce commands

$MODEL='Helsinki-NLP/opus-mt-en-ru'
$OUT='temp/model25-repro'
$RECIPES='examples/recipes/Helsinki-NLP_opus-mt-en-ru/cpu/cpu'

winml build -c "$RECIPES/translation_fp32_encoder_config.json" -m $MODEL -o "$OUT/encoder-fp32"
winml build -c "$RECIPES/translation_fp32_decoder_config.json" -m $MODEL -o "$OUT/decoder-fp32"
winml build -c "$RECIPES/translation_fp16_encoder_config.json" -m $MODEL -o "$OUT/encoder-fp16"
winml build -c "$RECIPES/translation_fp16_decoder_config.json" -m $MODEL -o "$OUT/decoder-fp16"

winml perf -m "$OUT/encoder-fp32/model.onnx" --ep cpu --device cpu
winml perf -m "$OUT/decoder-fp32/model.onnx" --ep cpu --device cpu
winml perf -m "$OUT/encoder-fp16/model.onnx" --ep cpu --device cpu
winml perf -m "$OUT/decoder-fp16/model.onnx" --ep cpu --device cpu

$env:WINMLCLI_RULES_DIR='rules'
winml analyze --model "$OUT/encoder-fp32/model.onnx" --ep all --output "$OUT/analyze-encoder-fp32.json"
winml analyze --model "$OUT/decoder-fp32/model.onnx" --ep all --output "$OUT/analyze-decoder-fp32.json"

winml eval --schema --task translation
python -m pytest tests/unit/eval/test_translation_evaluator.py tests/unit/models/winml/test_composite_from_pretrained.py tests/unit/eval/test_recipes.py tests/unit/recipes --tb=short --no-cov -q --color=no -m "not e2e and not npu and not gpu"
python -m ruff check src/ tests/
python -m mypy -p winml.modelkit

[ssss141414]

APPROVE

No blocking findings.

Reviewed exact candidate 3f5f321e4721e3022f7f50514e8cfcfb9ad4570d (tree ddf4c7810f2029937798e3ca1c03e4e6595a07af, base be3e59dd412d4918c5a852aa4d8f0207c34aaf6f), all 12 files, and the three-commit topology. The translation evaluator is generalized and data-driven: schema/registry/API dispatch, distinct dataset and tokenizer language semantics, optional source prefix, graph-capacity bounds, static batch-one generation, corpus TorchMetrics SacreBLEU-13a and chrF2 on a 0-100 scale, exact accounting, propagated runtime failures, and zero-valid-row failure are sound. The encoder-decoder repair right-pads only the encoder cross-attention mask before cache-oriented left-padding of decoder feeds. Cache progression, prompt prefill, generation limits, and sibling behavior are covered. No model/dataset hardcoding, dependency/README change, duplicate recipe, or scratch leakage is present.

All 8 authoritative manifests and 378 members rehashed exactly. Preserved L0/L1 reuse is valid after a 38-file artifact rehash and no-impact proof. Four artifacts have the claimed signatures, opset 17, realized FLOAT/FLOAT16 dtypes, colocated external data, and memory-bearing perf: FP32 encoder/decoder 69.23/22.51 ms; FP16 89.09/24.92 ms. Analyze is honest partial success: encoder 197/15, decoder 378/20; QNN partial Expand/Cast/Concat, ScatterND unknown for QNN/OpenVINO/TensorRT RTX, and absent rule sets are not runtime claims.

Independent L2 rerun reproduced FP32 encoder/decoder cosine 0.9999999999993413/0.9999999999986515, FP16 0.999998659921451/0.9999992557247197, all max-abs bounds, and exact full greedy token/text parity for two distinct sources. Independent pinned FLORES L3 rerun reproduced attempted/evaluated/skipped 2/2/0, distinct translations, SacreBLEU-13a 9.7950, and chrF2 56.8509; this is functional smoke only, not representative quality.

Reviewer reran focused 72, full Ruff, mypy 443, and all non-hardware partitions: Analyze 1529, models 1542, optim 876, commands 3706, remaining 964. PR #1394 is OPEN draft, exact author/branch/head/base/tree, mergeable, label model-scale-by-skill, exact UTF-8 body length 19,849 with Cyrillic preserved, and 12 files. Enumerated 0 line comments, 0 conversation comments, 0 reviews, 0 review threads, and 0 open threads.

Exact-head GitHub checks:

• Analyze (Python): COMPLETED/SUCCESS
• CodeQL: COMPLETED/SUCCESS
• license/cla: COMPLETED/SUCCESS
• lint: COMPLETED/SUCCESS
• test (analyze): COMPLETED/SUCCESS
• test (commands): COMPLETED/SUCCESS
• test (models): COMPLETED/SUCCESS
• test (optim): COMPLETED/SUCCESS
• test (remaining): COMPLETED/SUCCESS

PR #1213 remains OPEN draft and conflicting at a74eb377a2d43468c047ea10649aa52faec48991 with 7 overlapping paths. If it or main changes first, route Planner to drop/dedupe the generalized evaluator commit, rebase the recipe/runtime commits, reassess equivalent mask handling, and rerun invalidated tests. Lane A #314 is sealed at 2f4c04fa; its remote branch head was independently verified. Coverage is full for CPU/cpu fp32 and fp16; deferred tuples: none.

Disposition: terminal skill-level APPROVE. Keep the PR draft. This is an ordinary conversation comment, not a GitHub Review state.