Skip to content

Propagate directory upload errors without progress - #48242

Open
Jens (JensWehner) wants to merge 4 commits into
Azure:mainfrom
JensWehner:fix/ml-propagate-upload-errors
Open

Jens (JensWehner) wants to merge 4 commits into
Azure:mainfrom
JensWehner:fix/ml-propagate-upload-errors

Conversation

@JensWehner

@JensWehner Jens (JensWehner) commented Jul 24, 2026 •

Copy link
Copy Markdown
Contributor

Directory upload futures are only observed when progress is enabled. Always observe them so worker failures propagate with show_progress=False.

Fixes #48269

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for your contribution Jens (@JensWehner)! We will review the pull request and get back to you soon.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).
8 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Ensures directory upload worker failures propagate when progress display is disabled.

Changes:

  • Observes all upload futures regardless of progress settings.
  • Adds a regression test for show_progress=False.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
_asset_utils.py Propagates background upload exceptions without progress.
test_asset_utils.py Tests error propagation in that path.

@achauhan-scc

Amit Chauhan (achauhan-scc) commented Jul 27, 2026 •

Copy link
Copy Markdown
Member

Jens (@JensWehner) - Can you please update change log as well?

@saanikaguptamicrosoft

Copy link
Copy Markdown
Member

The CI is fixed in this PR: #47787

Once this is merged, pls rebase.

Copilot AI review requested due to automatic review settings July 27, 2026 06:42
@JensWehner
Jens (JensWehner) force-pushed the fix/ml-propagate-upload-errors branch from 372b79d to 9df430e Compare July 27, 2026 06:42

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings July 27, 2026 06:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

@github-actions

Copy link
Copy Markdown
Contributor
[Pilot] PR Pipeline Failure Analysis

A CI pipeline failed on this pull request. Here is an automated analysis of what went wrong and how to get the build green.

What failed

Three test failures were detected in the azure-ai-ml package across multiple CI environments:

  • macOS 3.11 (sdist): 2 unit test failures in TestPipelineJobEntity
    • test_automl_node_in_pipeline_image_multilabel_classification[single]
    • test_automl_node_in_pipeline_image_multilabel_classification[sweep]
  • Ubuntu 24.04 3.13 (whl): 1 e2e test failure in TestFeatureSet
    • test_create_and_get
  • Ubuntu 24.04 3.10 coverage (sdist): 1 unit test failure in TestLocalJobInvoker
    • test_serialize_patch_no_op

The two test_automl_node_in_pipeline_image_multilabel_classification failures and test_serialize_patch_no_op are unit test failures and likely reflect a code change that needs fixing. The test_create_and_get e2e test may be an infrastructure/environment issue.

Recommended next steps

  • Reproduce locally: run pytest sdk/ml/azure-ai-ml/tests/pipeline_job/unittests/test_pipeline_job_entity.py -k "test_automl_node_in_pipeline_image_multilabel_classification" and pytest sdk/ml/azure-ai-ml/tests/job_common/unittests/test_local_job_invoker.py -k "test_serialize_patch_no_op" to confirm and debug the unit test failures.
  • Review recent changes to AutoML node / image classification pipeline job entities and local job invoker serialization logic for regressions.
  • For the e2e test_create_and_get failure, consider re-running the pipeline — it may be a transient infrastructure issue.
  • See the CI troubleshooting guide: https://aka.ms/ci-fix
  • Push new commits to address the failures; this comment updates automatically on the next failing run.
Raw pipeline analysis (azsdk ci analyze)
Analyzing pipeline https://github.com/Azure/azure-sdk-for-python/pull/48242...
--------------------------------------------------------------------------------
Failed Tests
--------------------------------------------------------------------------------
{
  "/tmp/6618664/LLM Artifacts - macos311 - 1/ml-azure-ai-ml-test-junit-sdist.xml": [
    "sdk.ml.azure-ai-ml.tests.pipeline_job.unittests.test_pipeline_job_entity.TestPipelineJobEntity.test_automl_node_in_pipeline_image_multilabel_classification[single]",
    "sdk.ml.azure-ai-ml.tests.pipeline_job.unittests.test_pipeline_job_entity.TestPipelineJobEntity.test_automl_node_in_pipeline_image_multilabel_classification[sweep]"
  ],
  "/tmp/6618664/LLM Artifacts - Ubuntu2404_313 - 1/ml-azure-ai-ml-test-junit-whl.xml": [
    "sdk.ml.azure-ai-ml.tests.feature_set.e2etests.test_feature_set.TestFeatureSet.test_create_and_get"
  ],
  "/tmp/6618664/LLM Artifacts - ubuntu2404_310_coverage - 1/ml-azure-ai-ml-test-junit-sdist.xml": [
    "sdk.ml.azure-ai-ml.tests.job_common.unittests.test_local_job_invoker.TestLocalJobInvoker.test_serialize_patch_no_op"
  ]
}

Copilot detected the failing pipeline and generated the analysis above. To have it attempt a fix automatically, reply with @copilot please fix the failing pipeline on this PR.

Generated by Pipeline Analysis - Next Steps · 20.2 AIC · ⌖ 6.35 AIC · ⊞ 6.6K · ◷

@github-actions

github-actions Bot commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

Hi Jens (@JensWehner). Thank you for your interest in helping to improve the Azure SDK experience and for your contribution. We've noticed that there hasn't been recent engagement on this pull request. If this is still an active work stream, please let us know by pushing some changes or leaving a comment. Otherwise, we'll close this out in 7 days.

@github-actions github-actions Bot added the no-recent-activity There has been no recent activity on this issue. label Oct 2, 2026
Copilot AI balanced review requested due to automatic review settings October 2, 2026 10:35
@github-actions github-actions Bot removed the no-recent-activity There has been no recent activity on this issue. label Oct 2, 2026
@JensWehner

Copy link
Copy Markdown
Contributor Author

I updated it again, but this is complete and can be merged from my side.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The changelog accidentally removes existing entries and the 1.35.0 release section.

Review effort: Balanced
Findings: 1 Low severity

Open (1)

Comment thread sdk/ml/azure-ai-ml/CHANGELOG.md Resolved
Fixed various bugs related to directory uploads, artifact cache validation, job downloads, and log streaming. Improved performance for job listing and error reporting for local job paths.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings October 2, 2026 10:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The targeted fix correctly addresses the reported failure mode and includes regression coverage.

Review effort: Balanced
Findings: None

Resolved since last review (1)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Community Contribution Community members are working on the issue customer-reported Issues that are reported by GitHub users external to the Azure organization. Machine Learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

azure-ai-ml: upload_directory silently swallows upload errors when show_progress=False

4 participants