Add manifest-driven E2E release evaluation

[fangyangci]

Summary

• Add opt-in --release evaluation for 50 model/task cases across 16 machine-specific EP/device targets, with a precision-selection JSON manifest and Markdown historical evidence links, including NVIDIA DML.
• Preserve existing selection defaults: --priority remains P0-P3 with all four enabled by default, and non-release single-model, custom-registry, build-only and baseline modes remain available.
• Correct unquantized recipe labels and fix artifact-selection, input-dtype, synthetic-calibration and large-model QDQ serialization regressions found during release validation.
• Refresh winml catalog to the 43 model/task pairs that passed perf on all 16 release targets, keeping the existing display and JSON export formats.

Release Contract

• Reject explicitly supplied --priority, --hf-model, --registry, --task, --group, --model-type, --build-only and --update-baseline only when --release is set. Default values are not conflicts.
• Resolve shared EP targets using a normalized adjacent <machine>/<EP>_<device> pair in --output-dir; unrelated ancestor names do not select a target.
• Run one selected precision per case using an exact recipe match or a config fallback. A fallback does not establish equivalence to the historical configuration.
• Preserve execution controls, list/continue/retry support and precision-suffixed result paths.

Target Availability

• Check locally discovered EPs before resolving the requested runtime device binding; do not acquire missing Catalog packages during preflight.
• Missing EPs, registration failures, or absent device types print [SKIP] and return exit code 0 before model selection or result-directory creation.
• Automatic target selection, list/list-json, build-only and baseline-update modes keep their existing routing. Invalid target arguments and unexpected detection errors remain errors.

Precision Labels

• Rename fp16 recipes with quant: null to fp32 without changing their configuration contents; update the manifest and displayed target labels accordingly.
• Retain historical result JSON, evidence paths and reference-precision labels. A current fp32 link can still point to a historical __fp16 directory.
• These labels describe recipe/config selection, not independently measured ONNX tensor dtypes or EP execution precision. Renamed cases use new result directories, so --continue does not reuse old directory names.

QNN Regression Fixes

• Recover exact ONNX artifact paths from ANSI-colored, Rich-wrapped build output. When parsing cannot identify the artifact, accept only a single task-matching cache candidate; never choose between precision/config variants by modification time. This prevents BEiT GPU from benchmarking a w8a16 artifact after its build reported FP32.
• Preserve the export adapter's input dtype instead of coercing integer attention masks to bool by name. Add actual bool support to dummy-tensor generation and ONNX I/O extraction. Existing SDPA normalization stays at the attention-call boundary. This fixes the input conversion chain that produced floating masks and an infinite quantization scale in Splinter.
• Generate synthetic calibration tensors on demand. Arrow stores only sample indices, and an independent Torch generator seeded by index provides deterministic replay/rewind without changing global RNG state. This avoids preallocating all of Nougat decoder's approximately 322 MiB samples and converting their complete tensor payload into Arrow.
• Add regression coverage for wrapped paths, ambiguous caches, bool inputs, adapter dtype preservation, lazy generation, indexed replay and RNG isolation. The optional keyword-only generator argument remains compatible with existing callers.

Large-Model QDQ Fix

• Fix static 16-bit QDQ opset conversion for ONNX models with external weights. Debugging the real reranker cache reproduced EncodeError: Failed to serialize proto after materializing 2,275,216,132 weight bytes.
• Load the compact graph without external tensor data, convert the default ONNX opset to 21 when needed, and restore external initializers removed by the converter after checking their dtype and shape. Create the QDQ configuration before loading the weights, avoiding conversion of an oversized protobuf.
• Add a real external-data quantization regression covering the conversion order, standard-domain QDQ nodes and the final ONNX checker. Update output-publication coverage to use a real ONNX input and verify that the input remains unchanged.

Catalog Refresh

• Replace the previous 66 catalog entries with exactly 43 (model_id, task) pairs selected from the 2026-09-14 evaluation results: each pair has a boolean perf.passed == true on all 16 machine-specific EP/device targets, including NVIDIA TensorRT ARM.
• Match exact model IDs and task strings; do not retain additional tasks merely because the model ID is present. This is a perf-based selection, not a claim that accuracy passed on every target.
• Keep the packaged version/models schema, entry fields, EP/device mapping format, CLI table columns, filtering options, and JSON output/save format unchanged.
• Reuse 22 matching catalog entries and add 21 entries. New entries use the observed model type and the minimum recorded ONNX size across the 16 targets, converted to MiB and rounded to one decimal in the existing size_mb field.

Validation

• Catalog commit 56dc59b7: 122 catalog CLI/unit tests passed. The actual CLI JSON output was verified to match the 43 fully passing model/task pairs exactly; editor diagnostics and whitespace checks passed.
• QDQ follow-up commit 2ff1732f: 14 affected static-quantization/publication tests passed; scoped Ruff, editor diagnostics and whitespace checks passed.
• Original-config BAAI/bge-reranker-v2-m3 / text-classification / w8a16 was rerun through run_eval.py on Intel AI Boost with OpenVINO NPU and passed. Original int32/bool inputs, sequence length 1024 and W8A16 were retained; the final rerun measured 514.429 ms mean latency. Generated result files belong to the separate artifacts workspace, not this code commit.
• After rebasing onto the latest remote branch (including its main merge and CodeQL fixes), the affected runner/export/config/dataset/calibration pytest scope completed with 1369 passed, 6 pre-existing skips and 1 pre-existing xfail.
• Scope: tests/unit/eval/test_run_eval_script.py, tests/unit/export, tests/unit/config, tests/unit/datasets, and tests/unit/quant/calibration.
• Ruff passed for all eight changed files; whitespace checks passed. Pylance verified compatibility of all 11 InputTensorSpec.to_tensor call sites.
• Isolated real-model checks before the final rebase retained the original precision, ten calibration samples, ten perf iterations and two warmups: Splinter w8a16 and both Nougat w8a8 components passed PyPI QNN 2.5.0 NPU perf. All 433 Splinter and 586 Nougat decoder quantization scales checked were finite and positive.
• Nougat decoder build completed in approximately 196 seconds (quantization: 38.6 seconds), replacing the original 1200-second timeout. ORT profiles confirmed actual QNN partition execution for Splinter and Nougat decoder, with CPU nodes still present; this is not a claim of all-NPU execution.
• Previously recorded NVIDIA BAT list-only checks resolved 50 release cases each for DML, MLAS and TensorRT RTX.

Limitations

• The Intel DML kredor/punctuate-all temporary-directory error (WinError 145) did not reproduce in local debugging or the original-config forced rebuild. Local DML perf passed on NVIDIA RTX 5070, not Intel GPU, so the Intel GPU failure record remains unchanged. No cleanup error suppression was added.
• The QDQ hardware retry used local OpenVINO Catalog 1.8.26.0; the original Intel report used 1.8.95.0. This confirms the original model configuration on an Intel NPU, not equivalence of the complete original hardware/software stack.
• A broader quantization test attempt exposed an existing FP16 test mock missing CopyFrom; that unrelated test/implementation was not changed or skipped. The full suite was not rerun for this QDQ follow-up.
• Regenerate configurations and rebuild artifacts affected by the old mask-dtype bug. Existing cached artifacts are not repaired in place. Lazy sampling remains deterministic, but its sample sequence is not identical to the old eager implementation and can change calibration parameters.
• For the QNN investigation, accuracy and the full release batch were not rerun; its hardware checks preceded the final rebase, with the combined pytest scope rerun afterward. QNN release records were not overwritten or relabeled as PASS. For the QDQ follow-up, only the original-config Intel OpenVINO NPU reranker case was replaced after a fresh successful run, retaining its true retry timestamp; temporary September 15 result directories were removed. Accuracy was not rerun.
• DeBERTa GPU still fails PyPI QNN graph finalization with error 6022 after rebuilding with the correct integer mask. The same rebuilt artifact passes Catalog QNN. No source-default change or silent fallback was added.
• ResNet/ConvNeXt NPU inference executes on QNN, but op tracing still fails because the native EP reports an ETW profiling-state issue and writes no CSV. The profiling failure check was not disabled or relaxed.
• Nougat encoder cold session creation remains expensive: its perf command took approximately 1128 seconds, close to the original 1200-second budget. No EP tuning was applied to conceal this risk.

QDQ Subgraph and CI Follow-up (2026-09-17)

• Commit ced1d1b restores external initializers recursively after compact-model opset conversion, matching each graph by its enclosing node outputs and attribute scope. Same-named weights in separate branches are preserved independently, with the existing dtype and shape checks retained.
• Add actual external-data quantization regressions for one- and two-level nested If subgraphs. Verify exact child-weight preservation, ONNX checker validity, and CPU execution of both branches; retain the conversion-before-weight-loading test.
• Include the complete test_quant_passes.py and test_quantizer.py files in the existing optim CI group. Replace the outdated FP16 test fixture with a real ModelProto, resolving the earlier CopyFrom mock limitation without changing FP16 runtime behavior or adding skips.
• Verification: both complete quantization test files pass (43 tests). The workflow-derived optim group passes with 926 passed, 16 existing skips and 1 existing xfail. Scoped Ruff, editor diagnostics and staged whitespace checks pass.
• No real-model accelerator or accuracy E2E was rerun for this follow-up.

[Qiong Wu (qiowu) (DingmaomaoBJTU)]

Reviewed the release manifest/target selection, explicit-argument conflicts, recipe matching and fallback, build-artifact recovery, download timeout supervision, lazy calibration/dtype changes, external-weight QDQ conversion/restoration, recipe relabeling and catalog update. No blocking findings identified. Locally, 45 focused runner tests passed in an isolated harness covering argument parsing, artifact-path extraction, failure classification and result paths/IO (native EP deduction stubbed; 306 other cases deselected; three harness warnings). Also verified 50 unique release model/task cases with 16 consistent targets, 43 catalog entries, and a clean git diff --check. All currently reported PR CI checks pass. Full hardware release execution, large-model QDQ and calibration suites were not rerun locally; historical accuracy/hardware claims remain scoped to the evidence described in the PR.