Skip to content

fix(e2e): avoid parsing stage logs as final artifact paths - #1427

Merged
fangyangci merged 4 commits into
mainfrom
fangyangci/fix-e2e-artifact-path-parsing
Sep 18, 2026
Merged

fangyangci merged 4 commits into
mainfrom
fangyangci/fix-e2e-artifact-path-parsing

Conversation

@fangyangci

Copy link
Copy Markdown
Contributor

Summary

  • Extract deployment artifacts only from Final artifact: or Existing artifact found:, preferring the final artifact over reused or intermediate outputs.
  • Preserve ANSI stripping and Rich-wrapped paths, separate captured stderr/stdout, and handle OSError/ValueError when checking candidate files.
  • Add regression coverage for stage output, invalid paths, cache fallback, output-stream boundaries, and final-artifact precedence. Apply the configured formatting hooks to the two touched files.

Root Cause

The wrapped-path reconstruction introduced in #1411 also parsed stage-level Artifact: lines. Those lines include a size suffix, such as ..._export.onnx (335.3 MB), so the .endswith(".onnx") stop condition never matched. Subsequent warning and Optimize output was appended to the candidate filename. On the affected Windows agent, Path.is_file() raised WinError 1113 after the build subprocess had already exited successfully.

Stage artifacts are not necessarily the deployment-ready model and must not be selected as a fallback for the final build summary.

Validation

  • Complete E2E runner unit tests after formatting: 364 passed, including 15 artifact-path cases.
  • Configured pre-commit hooks passed for both changed files; editor diagnostics and git diff --check passed.
  • Real local Windows E2E reruns of openai/clip-vit-base-patch32 with QNN package 2.2480.49.0, matching the version reported in the failing logs:
Task Target Runner job labels Result
feature-extraction QNN NPU fp32, w8a16 2/2 PASS; runner exit 0
zero-shot-image-classification QNN NPU w8a8, w8a16 2/2 PASS; runner exit 0
zero-shot-image-classification QNN GPU default 1/1 PASS; runner exit 0

All three runs used --eval-type perf --timeout 1800 --raw-output with isolated temporary output/cache directories. Builds exercised the stage-output parsing path, and all eight component perf subprocesses exited 0. No path-parser exceptions occurred. Saved result JSON confirms all five jobs passed.

This validates the perf workflows from the three reported failures; accuracy was not rerun. The portable unit regression uses actual Rich output and injects the affected Windows filesystem exception; the hardware E2E reruns use real build/perf subprocesses. Temporary models, raw logs, and historical evaluation results are not part of this PR.

@fangyangci
fangyangci requested a review from a team as a code owner September 17, 2026 09:43
Comment thread scripts/e2e_eval/run_eval.py Fixed
fangyangci and others added 2 commits September 17, 2026 18:23
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
@fangyangci
fangyangci enabled auto-merge (squash) September 18, 2026 02:39
@fangyangci
fangyangci merged commit 68182e4 into main Sep 18, 2026
9 checks passed
@fangyangci
fangyangci deleted the fangyangci/fix-e2e-artifact-path-parsing branch September 18, 2026 03:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants