Repository navigation
[release/17.0] Harden KinD/Helm/kubectl and Radius install downloads in Kubernetes E2E tests - #22050
Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
Conversation
…2E tests (#21962) `DeployRadiusContainerToKind` and `DeployRadiusDotnetProjectsToKind` failed on PRs because downloads from GitHub release assets failed for a short time. Two runs show this. In [run 42468444499](https://github.com/microsoft/aspire/actions/runs/42468444499) (attempts 1 and 2), every Radius KinD test failed in `InstallKindAndHelmAsync`. The terminal recordings show `Retry 1/2/3: KinD download failed` across about 15 seconds, then `ERR`. In the same attempt, most of the 11 failing `KubernetesDeploy*` jobs failed the same way, and several others only recovered on their second or third retry. In [run 42581748484](https://github.com/microsoft/aspire/actions/runs/42581748484), the KinD download failed the same way, and a retry hit `rad install kubernetes` → `failed to fetch https://github.com/projectcontour/helm-charts/releases/download/contour-0.1.0/contour-0.1.0.tgz : 504 Gateway Time-out`. The cause is in the KinD/Helm/kubectl download loop. It used `curl -sSL` without `--fail`. On an HTTP error page curl exits 0, so the error never showed up. The loop only checked that the binary ran, and its three attempts 5 seconds apart covered about 15 seconds, which is shorter than these CDN error bursts. `rad install kubernetes` had no retry at all. The fix is in `tests/Aspire.Cli.EndToEnd.Tests/Helpers/KubernetesDeployTestHelpers.cs`: - **Tool downloads:** A new `GetToolDownloadCommand` builds the KinD, Helm and kubectl downloads. It treats any non-200 status as a failure and logs it (`HTTP <code>`), so the next failure is diagnosable. It makes 6 attempts with linear backoff (10/20/30/40/50s), cleans up partial downloads, and checks that the binary runs. Helm is now downloaded to a file before extraction instead of being piped to `tar`, and its binary is checked too. Step timeouts went from 90s to 9 min, the worst case for this retry budget. - **Radius control-plane install:** `rad install kubernetes` now retries up to 3 times, but only when the output shows a Helm chart download error (`error downloading helm chart|failed to fetch`). Install and readiness failures are not retried. Retries use `--reinstall`. rad installs the Radius chart before it fetches Contour ([`InstallRadius`](https://github.com/radius-project/radius/blob/v0.60.2/pkg/cli/helm/cluster.go)). A plain rerun would print "Found existing Radius installation", exit 0, and leave Contour uninstalled. `--reinstall` upgrades Radius and installs the missing Contour release ([`ApplyHelmChart`](https://github.com/radius-project/radius/blob/v0.60.2/pkg/cli/helm/helmaction.go)). Addresses the Radius KinD flake tracked on #18747. That issue also proposes re-pinning the container image off `dotnet/samples:aspnetapp`. That change is separate and not part of this PR, so the issue should stay open for it. ## Confidence - **Root cause:** high. All 9 recordings I analyzed from the failing Radius jobs (runs 42468444499 and 42581748484) end at the KinD download after 3 retries or at the Contour chart 504. Kubernetes deploy tests in the same attempt hit the same signature. - **Fix:** medium-high. I ran the generated shell locally: - KinD and kubectl downloaded from the real URLs. - The Helm tarball path worked against a local server. `get.helm.sh` is blocked in my sandbox. - A no-op when the tool is already on PATH. - A 404 URL failed after 6 attempts with `HTTP 404` logged. - A fake `rad` showed the retry passes `--reinstall` only on retries and does not retry non-download errors. The end-to-end KinD runs can only be checked in CI. A CDN outage longer than about 2.5 minutes would still fail. **How we stop this recurring:** Kubernetes tool downloads now fail on HTTP errors, log the status code and retry across a multi-minute window. The Radius chart install now recovers from transient GitHub 5xx responses instead of failing the job. ## Failure stats (14d, attempt 1) Window: the most recent 1,000 `ci.yml` pull_request runs (back to 2026-10-31). I checked attempt-1 jobs of every failed, cancelled or re-run run. | Job | Attempt-1 failures | Cause | |---|---|---| | `Cli.EndToEnd-RadiusDeployTests` | 3 | 2 GitHub download failures ([42468444499](https://github.com/microsoft/aspire/actions/runs/42468444499), [42581748484](https://github.com/microsoft/aspire/actions/runs/42581748484)), 1 runner shutdown ([42599265012](https://github.com/microsoft/aspire/actions/runs/42599265012)) | | `Cli.EndToEnd-RadiusDotnetProjectDeployTests` | 2 | GitHub download failures ([42468444499](https://github.com/microsoft/aspire/actions/runs/42468444499), [42581748484](https://github.com/microsoft/aspire/actions/runs/42581748484)) | The rate is low: 5 failures in roughly 2,000 attempt-1 job executions. Every failure that wasn't an infrastructure problem had the same download signature. When it happens, it fails every KinD-based test in the job and across the run. **Verdict: MODERATE.** The signature is consistent and clearly tied to the fixed code path, but the sample is small. ## Checklist - Is this feature complete? - [x] Yes. Ready to ship. - Are you including unit tests for the changes and scenario tests if relevant? - [x] No. These are E2E test-infrastructure changes. CI validates them by running the KinD/Radius E2E jobs. - Did you add public API? - [x] No - Does the change make any security assumptions or guarantees? - [x] No **🤖 Posted by AspireAgent (AI Agent)** --------- Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 07cf932)
14 tasks
This comment has been minimized.
This comment has been minimized.
Mitch Denny (mitchdenny)
marked this pull request as ready for review
November 10, 2026 05:15
Member
Author
|
Approved for inclusion in release/17.0. Do not merge yet. |
Mitch Denny (mitchdenny)
requested review from
James Newton-King (JamesNK),
Adam Ratzman (adamint),
David Negstad (danegsta),
David Fowler (davidfowl),
Eric Erhardt (eerhardt),
Ella Hathaway (ellahathaway),
Karol Zadora-Przylecki (karolz-ms),
Ankit Jain (radical) and
Sébastien Ros (sebastienros)
as code owners
November 10, 2026 08:08
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-52-pr21962
branch
from
November 10, 2026 08:09
9a6848c to
2d0ff4b
Compare
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 22050Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 22050" |
This comment has been minimized.
This comment has been minimized.
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-52-pr21962
branch
from
November 10, 2026 08:23
2d0ff4b to
9a6848c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #21962 to
release/17.0(cherry-picked from 07cf932 with no modifications).Layer 52 of 55 in the release/17.0 backport stack; stacked on #22049.
Backport recommendation: No — Test flake.
Validation: Stable patch IDs match. Builds/tests not run for this unchanged cherry-pick.
Backport of #21962
Checklist
<remarks />and<code />elements on your triple slash comments?