fix(build): honor --no-optimize and add multilingual similarity recipes

[ssss141414]

Summary

This L2 contribution adds CPU FP32 and FP16 sentence-similarity recipes for dell-research-harvard/lt-wikidata-comp-multi and fixes shared build-stage control for Hugging Face and direct-ONNX inputs. The candidate reached the committed L3 goal with full planned CPU coverage: both precisions passed structural, performance, and bounded numeric validation, and FP32 passed the bounded multilingual functional smoke Eval. A test-only follow-up now proves that configured compile returns a distinct output and that finalization consumes it in both concrete no-optimize paths. The current candidate is 3cd22269dfbfaac8baef63cb221c44c9e803af91, with parent a4ad838ecb83d103ee2a338dda5746a72492262b.

Model metadata

What the model does

A multilingual text-embedding checkpoint for company-alias record linkage and sentence similarity across 12 advertised languages; it exports contextual token states consumed by downstream mean pooling into 768-dimensional embeddings.

• Evidence: pinned checkpoint dell-research-harvard/lt-wikidata-comp-multi@129b41d6e02494453aef06108baec55d85c6ad8c, its model card, config.model_type=xlm-roberta, architectures=[XLMRobertaModel], and 1_Pooling/config.json with pooling_mode_mean_tokens=true.
• Confidence: verified.

Primary user stories

• A user supplies multilingual company names or descriptions to obtain embedding similarity for record linkage, deduplication, clustering, or semantic search. Evidence: the pinned checkpoint model card. Confidence: verified.

Supported tasks

• sentence-similarity on the checkpoint and WinML surfaces. Evidence: Hub pipeline_tag=sentence-similarity; winml eval --schema --task sentence-similarity exits 0. Confidence: verified.
• feature-extraction on Transformers, Optimum ONNX, and WinML surfaces. Evidence: winml inspect resolves feature extraction through XLMRobertaIOConfig; current main contains exact-model FP32/FP16 feature-extraction recipes. Confidence: verified.

Model architecture

XLMRobertaModel
|- Embeddings (250002 vocab, position + token type, width 768)
|- Encoder stack x 12
|  |- Bidirectional self-attention (12 heads)
|  |- Feed-forward (768 -> 3072 -> 768, GELU)
|  `- Residual + LayerNorm
|- Exported output: last_hidden_state [1, sequence, 768]
`- Downstream sentence-transformers mean pooling -> 768-d embedding

• Source/confidence: exact pinned config dimensions, locked Transformers 5.14.1 modeling_xlm_roberta.py, pinned pooling config, and built ONNX graph structure (verified).

Validation and support evidence

Baseline

Baseline was freshly measured on main commit 6b16162816121bec4f1610726dc0769b2df5a703 with winml, version 0.0.1.dev0. Auto-config generated a sentence-similarity recipe. A no-optimize CPU build exited 0 in 124.8 s (Export 69.0 s, Optimize 30.0 s), but still emitted an Optimize stage and optimized artifact, proving that --no-optimize was ignored by the concrete HF pipeline. The resulting baseline perf was 310.26 ms mean, 312.81 ms p50, 314.20 ms p90, 3.22 samples/s, and +105.0 MB RAM over three measured iterations. Baseline Eval evidence contained only Python REPL startup and established no task metric.

The Optimum registry probe found the vendor and WinML surfaces both advertising feature extraction, fill-mask, multiple choice, question answering, text classification, and token classification; WinML added no task. The aggregate probe exceeded its 10-minute wall-time and was terminated (probe_exit_code=-1), so no probe PASS is inferred. The baseline goal floor was L1.

Goal

• Effort: L2.
• Goal ceiling: L3, reached with PASS.
• Outcome: L2.
• Success definition: no-recipe XLM-R build acceptance; concrete HF and ONNX --no-optimize behavior with quantization and defaults preserved; independent FP32/FP16 CPU recipe validation with true-FP16 evidence; and one final-SHA, two-row, pinned English-German functional smoke Eval with all fan-out caps retained.

No charter re-issue or ceiling change occurred.

Outcome

The shipped tier is L2 and the highest goal verdict is L3 PASS. Coverage is full for the planned CPU/cpu FP32 and FP16 tuples, with no deferred tuples. The contribution adds two sentence-similarity recipes, changes shared build behavior, and adds concrete-sink regression tests.

The follow-up from a4ad838ecb83d103ee2a338dda5746a72492262b to 3cd22269dfbfaac8baef63cb221c44c9e803af91 changes only tests/unit/commands/test_build.py. Fresh testing at 3cd22269... covered the four concrete HF/ONNX sink cases, the complete commands partition, and touched-file Ruff. Because production code, recipes, uv.lock, model artifacts, evaluator, compiler, optimizer, and workflows are unchanged, the Tester made an explicit REUSE-REBIND decision for L0 builds/structure, L1 perf, L2 parity, L3 Eval, component/op Analyze, models/optim/remaining partitions, full Ruff, and mypy. Those values below were executed at a4ad838e... and are preserved as provenance-bound evidence; they were not rerun or relabeled as executions at 3cd22269....

Learner findings xlm-roberta-005 through xlm-roberta-010 and methodology finding _meta-114 were published separately in Lane A PR gim-home/ModelKitArtifacts#289 at commit 6b485ff17aeb128651335f0d53301479b06f3151, based on prerequisite PR #288 at commit 63e0bb74b4b79a394d5aaffef496d3d83404447b. Those skill and knowledge files are not part of this winml-cli contribution.

Per-EP/device/precision results and Functional smoke Eval

The following L0-L3 model values are REUSE-REBIND evidence executed at a4ad838e...; no production or artifact-producing input changed in the test-only follow-up.

Tier	EP / Device	Precision	Verdict	Mean	p50	Throughput	RAM delta	Precision evidence
L0	CPUExecutionProvider / cpu	fp32	PASS	-	-	-	-	383 nodes; 200 FLOAT initializers; 1,109,820,432-byte external data; FLOAT output
L0	CPUExecutionProvider / cpu	fp16	PASS	-	-	-	-	384 nodes; 200 FLOAT16 initializers; 554,914,320-byte external data; FP32 I/O preserved; FP16/FP32 size ratio 0.5000036978955168
L1	CPUExecutionProvider / cpu	fp32	PASS	424.85 ms	373.55 ms	2.35 samples/s	+105.0 MB	printed output precision float32
L1	CPUExecutionProvider / cpu	fp16	PASS	314.49 ms	316.34 ms	3.18 samples/s	+122.0 MB	requested FP16; printed output precision float32 because I/O is preserved; true FP16 artifact evidence above

The L1 figures are short-run measurements with one warmup and three measured iterations; they are not a stable benchmark claim.

L2 artifact	Verdict	Frozen bounds	Observed values
fp32	PASS	minimum cosine 0.9999; maximum absolute error 0.0005	minimum cosine 0.9999999403953552; maximum absolute error 1.2293457984924316e-07
fp16	PASS	minimum cosine 0.999; maximum absolute error 0.02	minimum cosine 0.9999995231628418; maximum absolute error 0.00013403967022895813

Functional smoke Eval: L3 PASS, executed at candidate a4ad838ecb83d103ee2a338dda5746a72492262b and retained for 3cd22269... by the Tester's explicit REUSE-REBIND decision. It used FP32 CPU and mteb/sts17-crosslingual-sts at requested and resolved revision bb1de64ee6fd9ccd66e0bdbadcafe990f652d887, config en-de, split test. Deterministic first-N selection requested and processed two English-German sentence pairs. Schema was verified; labels are STS relatedness scores; predictions are cosine similarities of attention-mask mean-pooled, L2-normalized last_hidden_state embeddings; pooling was verified against the pinned configuration. Fan-out was capped at two sentence pairs, one en-de language direction, sequence length 512, no beams, and no candidate labels or prompts. The raw spearman_correlation was 0.9999999999999999 (approximately 1.0). With only two rows, this is functional end-to-end operability evidence only, not representative accuracy or benchmark-quality evidence.

Delta

Changed winml-cli files across the full current candidate:

• src/winml/modelkit/commands/build.py
• tests/unit/commands/test_build.py
• examples/recipes/dell-research-harvard_lt-wikidata-comp-multi/cpu/cpu/sentence-similarity_fp32_config.json
• examples/recipes/dell-research-harvard_lt-wikidata-comp-multi/cpu/cpu/sentence-similarity_fp16_config.json

The FP32 recipe is identical to the Planner's starting auto-config. The FP16 recipe changes only JSON pointer /quant, from null to explicit FP16 configuration: mode fp16, samples 10, calibration method minmax, weight and activation types uint8, per_channel=false, symmetric=false, uniform distribution, model/task/type identity, and fp16_keep_io_types=true; optional symmetry, paths, op lists, block lists, and seed remain null. The production recipe README is untouched. Recipe-free acceptance passed, and the delta is reducibility-consistent with the charter.

The repair commit changes only tests/unit/commands/test_build.py: its HF and ONNX no-optimize tests now configure compile to return a path distinct from the quantized or exported input, assert compile ordering, and assert that finalization consumes that compiled path. It changes no production behavior, recipes, lock data, or artifacts.

Bug fix explanation

1. Symptom and trigger: winml build ... --no-optimize propagated the flag but still ran Optimize on a concrete Hugging Face build; direct-ONNX builds had the same sink-level gap.
2. Root cause: the command layer placed skip_optimize in build_pipeline_extra_kwargs, but _build_hf_pipeline and _build_onnx_pipeline did not consume it before unconditionally invoking the optimize helper. Existing propagation-only tests mocked the concrete sink and could not detect this.
3. Mechanism: _build_hf_pipeline and _build_onnx_pipeline consume skip_optimize and bypass only the Optimize helper. Requested FP16/other quantization, compile, and finalize stages continue to run.
4. General rule: the fix is driven by the existing stage-control flag at both concrete pipeline sinks. It contains no model ID, checkpoint, or XLM-R special case.
5. Compatibility: default builds still optimize; skipping optimization does not suppress requested quantization; lower HF/ONNX semantics, recipe behavior, compile, and finalization are preserved. There are no intentional behavior changes beyond making --no-optimize honor its documented meaning.
6. Regression evidence: at current candidate 3cd22269..., four concrete HF/ONNX sink cases passed in 4.83 s. They cover quantized and no-quant paths, make compile output distinct, assert compile receives the quantized or exported output, and assert finalization receives the compiled output while Optimize alone is skipped. The complete commands partition passed with 3648 passed, 9 skipped, 1 warning in 317.31 s (0:05:17), and touched-file Ruff reported All checks passed!.

Analyze summary - component level and op level

ANALYZE-PARTIAL-SUCCESS: Analyze emitted complete seven-EP JSON and exited 1; the nonzero status is retained because findings include partial/unknown support and EPs without rule data. This is static rule analysis of the FP32 graph, not runtime execution. No all-EP runtime support was measured, and no FP16 all-EP Analyze run was supplied. This section is REUSE-REBIND evidence executed at a4ad838e..., not a fresh Analyze run at 3cd22269....

Component-level summary

Artifact	Architecture coverage	Mapping	Confidence / unresolved gaps
fp32	embeddings; 12 encoder layers with self-attention and MLP; last_hidden_state boundary; downstream mean pooling	383 mapped, 0 unmapped	hierarchy tags plus frozen tensor-boundary mapping; output is a tensor boundary, pooling is outside the graph, and optimized Gemm nodes cannot be uniquely split between attention and MLP

Op-level summary

Artifact	Graph	Dominant ops	Actionable static EP findings
fp32	383 operators / 19 types	Reshape 121; Gemm 72; Transpose 48; Add 39; LayerNormalization 25; MatMul 24; Mul 13; Softmax 12	QNN GPU: partial Gather and GatherElements, unknown Where; NvTensorRTRTX/OpenVINO GPU: unknown Where

CUDA and MIGraphX classifications had ambiguous type-derived counts (380 supported nodes, 2 partial, 1 unknown). TensorRT and DML had no rule data, leaving all 383 nodes unclassified. None of these static rows claims provider runtime execution.

CI quality partitions

Fresh local evidence at 3cd22269...:

• Focused concrete HF/ONNX sink partition: 4 passed in 4.83 s.
• Commands partition: 3648 passed, 9 skipped, 1 warning in 317.31 s (0:05:17).
• Touched-file Ruff: All checks passed.

Live PR #1370 CI at head 3cd22269dfbfaac8baef63cb221c44c9e803af91: all nine reported checks completed successfully: Analyze (Python), lint, test (analyze), test (models), test (optim), test (commands), test (remaining), CodeQL, and license/cla.

Preserved REUSE-REBIND evidence executed at a4ad838e...:

• insert-license: Passed.
• ruff-full: All checks passed.
• mypy-full: Success, no issues in 441 source files.
• Analyze partition: 1529 passed, 45 skipped.
• Models partition: 1538 passed, 6 skipped, 2 xfailed.
• Optim partition: 876 passed, 16 skipped, 1 xfailed.
• Remaining partition: 936 passed, 2 skipped, 1 deselected, 2 warnings in 559.69 s (0:09:19).
• Compatibility-focused partition: HF/ONNX no-optimize, quant preservation, and defaults passed.

The prior remaining-partition attempt was environment-blocked because Node 22 was absent; the final supplemental run used Node v22.23.2 on PATH and passed.

Reproduce commands

Run from a winml-cli checkout after installing its frozen environment. These are the Tester-supplied portable public commands retained from the original model-validation handoff:

$OUT='temp/lt-wikidata-comp-multi-test'
winml build -c examples/recipes/dell-research-harvard_lt-wikidata-comp-multi/cpu/cpu/sentence-similarity_fp32_config.json -m dell-research-harvard/lt-wikidata-comp-multi -o $OUT/fp32
winml build -c examples/recipes/dell-research-harvard_lt-wikidata-comp-multi/cpu/cpu/sentence-similarity_fp16_config.json -m dell-research-harvard/lt-wikidata-comp-multi -o $OUT/fp16 --precision fp16
winml perf -m $OUT/fp32/model.onnx --ep cpu --device cpu --warmup 1 --iterations 3
WINMLCLI_RULES_DIR=<rules-root> winml analyze --model $OUT/fp32/model.onnx --ep all --output $OUT/analyze.json

This body is ready to replace the description of existing draft PR #1370, which carries the model-scale-by-skill label. This refresh does not mutate the PR.

[ssss141414]

REQUEST_CHANGES

Reviewed candidate a4ad838ecb83d103ee2a338dda5746a72492262b against PR base main at 95667fe5f8b52b53dfc8fa233d37bbf92048b351.

One required _meta-114 regression guarantee is not proved:

• Producer/Tester: In tests/unit/commands/test_build.py, both test_hf_skip_optimize_preserves_quantize and test_onnx_skip_optimize_preserves_quantize install mock_compile, but never assert that it is called and return the incoming current_path unchanged. The tests therefore cannot distinguish “compile ran after quantization” from “compile was silently skipped,” and the final-copy assertions do not prove that finalization consumes compiled output. Configure a non-null compile section, return a distinct compiled path, assert _run_compile_stage is called once with the quantized/exported path for both quant_mode=None and quant_mode='fp16', and assert finalization copies the compiled result. Rerun the affected focused tests and commands partition.

The implementation itself consumes skip_optimize at both HF and direct-ONNX sinks without model-ID branching, and the existing tests do prove Optimize is skipped, quantization remains reachable, final copy occurs, and defaults still optimize. The missing compile assertion is nevertheless an explicit blocking requirement of _meta-114, not a style preference.

Evidence review: all six supplied upstream manifests rehashed cleanly; the candidate worktree is clean and matches parent/tree identity; the PR is OPEN and DRAFT with model-scale-by-skill; its body matches the sealed Explainer report; the exact diff is the two CPU recipes plus build.py and test_build.py; Lane A PR refs #288/#289 resolve to the cited commits; and the Reviewer rerun passed all 7 current concrete-sink cases. Complete pagination found 2 conversation comments, 0 line comments, 0 GitHub Reviews, 0 review threads, and 0 open threads.

Live checks at capture time:

• Analyze (Python): COMPLETED / SUCCESS
• lint: COMPLETED / SUCCESS
• test (analyze): COMPLETED / SUCCESS
• test (models): COMPLETED / SUCCESS
• test (optim): COMPLETED / SUCCESS
• test (commands): COMPLETED / SUCCESS
• test (remaining): COMPLETED / SUCCESS
• CodeQL: COMPLETED / SUCCESS
• license/cla: COMPLETED / SUCCESS

Active checks above are recorded as pending, not PASS. A fresh Reviewer pass must rebind the new head, re-enumerate all threads/comments, and require every live check to complete successfully.

This is an ordinary reviewer opinion comment, not a GitHub Review state. Preserve the draft PR.

[ssss141414]

APPROVE

Reviewed repaired candidate 3cd22269dfbfaac8baef63cb221c44c9e803af91 (parent a4ad838ecb83d103ee2a338dda5746a72492262b, tree 95c7e0ae29317c2bcfdf21828f1440c81a52a56d) against PR #1370 base main.

The prior _meta-114 REQUEST_CHANGES is resolved. The concrete HF and direct-ONNX no-optimize tests now use distinct exported/copied, quantized, and compiled paths; parameterize both no-quant and FP16; assert compile receives the prior stage output; assert finalization consumes the compiled output; prove Optimize alone is skipped; and retain default optimization coverage. Independent rerun: 6 focused cases passed.

Engineering review found the combined four-file diff sound and scoped to Outcome L2. build.py consumes skip_optimize at both concrete sinks, preserves config-driven pre-quantized skipping, and leaves quantize, compile, and finalize stages enabled. There is no model-specific branch or default regression. Both CPU recipes match the model contract; FP32 is auto-config-identical and FP16 differs only by explicit FP16 quantization. Production README is unchanged.

Evidence verification: all nine frozen Planner/Producer/Tester/Learner/prior-Reviewer/repair/Explainer manifests and members rehashed exactly. Candidate identity is clean with no drift. Direct evidence confirms FP32/FP16 L0-L3 coverage, true FP16 initializers and 0.500004 external-data ratio, complete 383-node/19-op static Analyze with retained limitations, bounded two-row en-de functional smoke semantics, and published Lane A PRs #288/#289 at the cited commits. The test-only repair changes no production, recipe, lock, artifact, evaluator, compiler, optimizer, or workflow input, so the explicit REUSE-REBIND of unaffected evidence is valid.

Independent quality gates at this head: full Ruff passed; mypy -p winml.modelkit passed for 441 source files; and the exact commands CI matrix passed with 3648 passed and 9 skipped. A discarded Reviewer probe referencing nonexistent tests/integration/commands is preserved internally and superseded by the exact workflow invocation.

Live PR state: OPEN draft, base main, head model11-producer/lt-wikidata-comp-multi-c4a8f210, label model-scale-by-skill, four expected files, sealed report body normalized-equal, 1 conversation comments before this publication, 0 line comments, 0 GitHub Reviews, 0 review threads, and 0 open threads.

Live checks at the reviewed SHA:

• Analyze (Python): COMPLETED/SUCCESS
• lint: COMPLETED/SUCCESS
• test (analyze): COMPLETED/SUCCESS
• test (models): COMPLETED/SUCCESS
• test (optim): COMPLETED/SUCCESS
• test (commands): COMPLETED/SUCCESS
• test (remaining): COMPLETED/SUCCESS
• CodeQL: COMPLETED/SUCCESS
• license/cla: COMPLETED/SUCCESS

Coverage is full: reachable and verified tuples are CPU/cpu FP32 and CPU/cpu FP16; deferred tuples: none.

Disposition: terminal skill-level APPROVE. This is an ordinary conversation comment, not a GitHub Review state. Preserve the draft PR.