Skip to content

[release/17.0] Harden KinD/Helm/kubectl and Radius install downloads in Kubernetes E2E tests - #22050

Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
mitchdenny-backport-17-0-51-pr21932from
mitchdenny-backport-17-0-52-pr21962
Closed

Mitch Denny (mitchdenny) wants to merge 1 commit into
mitchdenny-backport-17-0-51-pr21932from
mitchdenny-backport-17-0-52-pr21962

Conversation

@mitchdenny

Copy link
Copy Markdown
Member

Description

Backport of #21962 to release/17.0 (cherry-picked from 07cf932 with no modifications).

Layer 52 of 55 in the release/17.0 backport stack; stacked on #22049.

Backport recommendation: No — Test flake.

Validation: Stable patch IDs match. Builds/tests not run for this unchanged cherry-pick.

Backport of #21962

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

…2E tests (#21962)

`DeployRadiusContainerToKind` and `DeployRadiusDotnetProjectsToKind`
failed on PRs because downloads from GitHub release assets failed for a
short time. Two runs show this. In [run
42468444499](https://github.com/microsoft/aspire/actions/runs/42468444499)
(attempts 1 and 2), every Radius KinD test failed in
`InstallKindAndHelmAsync`. The terminal recordings show `Retry 1/2/3:
KinD download failed` across about 15 seconds, then `ERR`. In the same
attempt, most of the 11 failing `KubernetesDeploy*` jobs failed the same
way, and several others only recovered on their second or third retry.
In [run
42581748484](https://github.com/microsoft/aspire/actions/runs/42581748484),
the KinD download failed the same way, and a retry hit `rad install
kubernetes` → `failed to fetch
https://github.com/projectcontour/helm-charts/releases/download/contour-0.1.0/contour-0.1.0.tgz
: 504 Gateway Time-out`.

The cause is in the KinD/Helm/kubectl download loop. It used `curl -sSL`
without `--fail`. On an HTTP error page curl exits 0, so the error never
showed up. The loop only checked that the binary ran, and its three
attempts 5 seconds apart covered about 15 seconds, which is shorter than
these CDN error bursts. `rad install kubernetes` had no retry at all.

The fix is in
`tests/Aspire.Cli.EndToEnd.Tests/Helpers/KubernetesDeployTestHelpers.cs`:
- **Tool downloads:** A new `GetToolDownloadCommand` builds the KinD,
Helm and kubectl downloads. It treats any non-200 status as a failure
and logs it (`HTTP <code>`), so the next failure is diagnosable. It
makes 6 attempts with linear backoff (10/20/30/40/50s), cleans up
partial downloads, and checks that the binary runs. Helm is now
downloaded to a file before extraction instead of being piped to `tar`,
and its binary is checked too. Step timeouts went from 90s to 9 min, the
worst case for this retry budget.
- **Radius control-plane install:** `rad install kubernetes` now retries
up to 3 times, but only when the output shows a Helm chart download
error (`error downloading helm chart|failed to fetch`). Install and
readiness failures are not retried. Retries use `--reinstall`. rad
installs the Radius chart before it fetches Contour
([`InstallRadius`](https://github.com/radius-project/radius/blob/v0.60.2/pkg/cli/helm/cluster.go)).
A plain rerun would print "Found existing Radius installation", exit 0,
and leave Contour uninstalled. `--reinstall` upgrades Radius and
installs the missing Contour release
([`ApplyHelmChart`](https://github.com/radius-project/radius/blob/v0.60.2/pkg/cli/helm/helmaction.go)).

Addresses the Radius KinD flake tracked on #18747. That
issue also proposes re-pinning the container image off
`dotnet/samples:aspnetapp`. That change is separate and not part of this
PR, so the issue should stay open for it.

## Confidence

- **Root cause:** high. All 9 recordings I analyzed from the failing
Radius jobs (runs 42468444499 and 42581748484) end at the KinD download
after 3 retries or at the Contour chart 504. Kubernetes deploy tests in
the same attempt hit the same signature.
- **Fix:** medium-high. I ran the generated shell locally:
  - KinD and kubectl downloaded from the real URLs.
- The Helm tarball path worked against a local server. `get.helm.sh` is
blocked in my sandbox.
  - A no-op when the tool is already on PATH.
  - A 404 URL failed after 6 attempts with `HTTP 404` logged.
- A fake `rad` showed the retry passes `--reinstall` only on retries and
does not retry non-download errors.

The end-to-end KinD runs can only be checked in CI. A CDN outage longer
than about 2.5 minutes would still fail.

**How we stop this recurring:** Kubernetes tool downloads now fail on
HTTP errors, log the status code and retry across a multi-minute window.
The Radius chart install now recovers from transient GitHub 5xx
responses instead of failing the job.

## Failure stats (14d, attempt 1)

Window: the most recent 1,000 `ci.yml` pull_request runs (back to
2026-10-31). I checked attempt-1 jobs of every failed, cancelled or
re-run run.

| Job | Attempt-1 failures | Cause |
|---|---|---|
| `Cli.EndToEnd-RadiusDeployTests` | 3 | 2 GitHub download failures
([42468444499](https://github.com/microsoft/aspire/actions/runs/42468444499),
[42581748484](https://github.com/microsoft/aspire/actions/runs/42581748484)),
1 runner shutdown
([42599265012](https://github.com/microsoft/aspire/actions/runs/42599265012))
|
| `Cli.EndToEnd-RadiusDotnetProjectDeployTests` | 2 | GitHub download
failures
([42468444499](https://github.com/microsoft/aspire/actions/runs/42468444499),
[42581748484](https://github.com/microsoft/aspire/actions/runs/42581748484))
|

The rate is low: 5 failures in roughly 2,000 attempt-1 job executions.
Every failure that wasn't an infrastructure problem had the same
download signature. When it happens, it fails every KinD-based test in
the job and across the run.

**Verdict: MODERATE.** The signature is consistent and clearly tied to
the fixed code path, but the sample is small.

## Checklist

- Is this feature complete?
  - [x] Yes. Ready to ship.
- Are you including unit tests for the changes and scenario tests if
relevant?
- [x] No. These are E2E test-infrastructure changes. CI validates them
by running the KinD/Radius E2E jobs.
- Did you add public API?
  - [x] No
- Does the change make any security assumptions or guarantees?
  - [x] No

**🤖 Posted by AspireAgent (AI Agent)**

---------

Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit 07cf932)
@github-actions

This comment has been minimized.

@mitchdenny
Mitch Denny (mitchdenny) marked this pull request as ready for review November 10, 2026 05:15
@mitchdenny

Copy link
Copy Markdown
Member Author

Approved for inclusion in release/17.0. Do not merge yet.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 22050

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 22050"

@github-actions

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants