Skip to content

fix(eval): respect image-to-text pipeline call contracts - #1405

Open
Yue Sun (KayMKM) wants to merge 1 commit into
mainfrom
kaymkm-fix-image-to-text-eval-pr
Open

Yue Sun (KayMKM) wants to merge 1 commit into
mainfrom
kaymkm-fix-image-to-text-eval-pr

Conversation

@KayMKM

Copy link
Copy Markdown
Contributor

Summary

  • Supply an empty text prompt only when the active pipeline explicitly declares that parameter. Legacy ImageToTextPipeline receives only the image; multimodal ImageTextToTextPipeline still receives its required empty prompt.
  • Raise an actionable error with the original cause when pipeline failures leave no evaluated predictions. Preserve partial results and empty-dataset behavior.
  • Cover both real Hugging Face pipeline call contracts using generated images/references, and cover all-sample failures. Document the behavior.
  • Keep the existing E2E CER/CIDEr thresholds, sample-count assertion, and no-skips assertion unchanged.

Background

The daily E2E run 156998365 fails TestEvalPerTask::test_image_to_text_fp16 on QNN and AMD. Every sample raises ImageToTextPipeline._sanitize_parameters() got an unexpected keyword argument 'text', and the evaluator returns cer=None after skipping them all. Fix the evaluator rather than relaxing the E2E assertions.
The independent AMD EP-discovery failures and OV agent disconnect are out of scope.

Validation

  • 46 focused pytest cases passed, covering evaluator behavior, text metrics, task mapping and evaluator/CLI error propagation.
  • Ruff passed for the changed Python files.
  • Full model/hardware E2E is left for manual execution on a host where the hardware-gated case executes rather than skips:
uv run --no-sync pytest tests\e2e\test_eval_e2e.py::TestEvalPerTask::test_image_to_text_fp16 -m e2e --timeout=1800

@KayMKM
Yue Sun (KayMKM) requested a review from a team as a code owner September 9, 2026 07:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant