Skip to content

[release/17.0] Give dotnet run apphost.cs the aspire run startup budget in SingleFileAppHostInit E2E test - #22041

Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
mitchdenny-backport-17-0-42-pr21894from
mitchdenny-backport-17-0-43-pr21967
Closed

Mitch Denny (mitchdenny) wants to merge 1 commit into
mitchdenny-backport-17-0-42-pr21894from
mitchdenny-backport-17-0-43-pr21967

Conversation

@mitchdenny

Copy link
Copy Markdown
Member

Description

Backport of #21967 to release/17.0 (cherry-picked from f591772 with no modifications).

Layer 43 of 55 in the release/17.0 backport stack; stacked on #22040.

Backport recommendation: No — Test flake.

Backport of #21967

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

…eAppHostInit E2E test (#21967)

The CLI E2E test
`SingleFileAppHostInitDotnetRunTests.AspireInitSingleFileAppHostRunsViaDotnetRunAppHost`
timed out waiting for `Press CTRL+C to stop the AppHost and exit.` after
`dotnet run apphost.cs` ([run
41004976623](https://github.com/microsoft/aspire/actions/runs/41004976623/job/123901814066),
[run
42093304445](https://github.com/microsoft/aspire/actions/runs/42093304445/job/127485097097),
[run
42093812755](https://github.com/microsoft/aspire/actions/runs/42093812755/job/127485652686)).
The terminal recordings show the cause. Before the AppHost starts, the
.NET file-based runner does a cold NuGet restore of `apphost.cs`. The
NuGet.config that `aspire init` generates only adds the local hive and
doesn't clear other sources, so transitive packages still come from
nuget.org. That restore normally finishes in under 10s, but in all three
failures it took 50–77s. The build and the AppHost start then took about
5s more, and the test's hard-coded 1-minute wait ran out first. The two
11-05 failures were on different branches but stalled within the same
minute, which points to a slow upstream feed rather than this repo. In
every failure the AppHost did reach "Press CTRL+C" a few seconds after
the wait expired.

This is the same restore, build and start work that `aspire run` does,
and every other E2E ready-wait for it uses
`CliE2EAutomatorHelpers.AspireRunReadyTimeout` (240s). The test now uses
that budget too. So a genuine failure doesn't wait the full 4 minutes,
the wait also fails immediately if `dotnet run` returns to an error
prompt. The change is test-only.

## Confidence

High that this is the cause and that the change fixes it. The evidence:
- **Recordings:** In the 3 failures, restore was still running when the
wait expired. In 28 passing recordings (11-01 to 11-06), `dotnet run
apphost.cs` printed "Press CTRL+C" 10.5–19.0s after it started.
- **Local reproduction:** I ran a cold `dotnet build apphost.cs` in the
.NET 10 SDK container, which is the phase that stalls. To model a slow
feed, I routed NuGet through a local HTTPS proxy that delays each
upstream read by 20ms. That reproduces CI's degraded restore time.

| Network | Old budget (60s) | New budget (240s) |
|---|---|---|
| Healthy (about 8s) | 5/5 pass | 5/5 pass |
| Slow feed (about 72s) | **0/5 pass** (all timed out) | **5/5 pass** |

The test project builds with 0 warnings. Main uncertainty: I couldn't
run the full E2E test locally, because it needs the PR's CLI archive and
the prebuilt E2E image. This PR's CI covers it. A feed that is slower
than 240s would still fail, as it would for every other `aspire run` E2E
test.

## How we stop this recurring

Waits that cover a cold NuGet restore, build and AppHost start should
use `CliE2EAutomatorHelpers.AspireRunReadyTimeout` instead of a
hand-picked timeout. That budget is shared, so it is tuned in one place.
After this change, no E2E test waits for "Press CTRL+C" with a budget
under 2 minutes. Pairing a long budget with a fail-fast check on the
error prompt keeps genuine failures quick.

## Failure stats (14d, attempt 1)

| Scope | Runs | Failures of this test | Rate |
|---|---|---|---|
| CI workflow, 2026-10-26 to 2026-11-09, first attempt | about 1,150
decided runs (158 on `main`) | 3 (0 on `main`): 10-30, 11-05 ×2 | about
0.26% |

All 3 had the same signature (`Step N failed — WaitUntilText("Press
CTRL+C to stop the AppHost and exit.") Timed out after 00:01:00`), and
all 3 passed on rerun. I left out 4 failures from
`danegsta-nuget-mapping-regression` (10-29), because that branch broke
NuGet package mapping and failed on every attempt. I found the failures
by querying this job's check run on every commit of a CI run that failed
or was rerun in the window (535 commits, 1,442 check runs).

**Verdict: MODERATE.** The signature and mechanism are consistent, and
the local reproduction is deterministic. But there are only 3 hits, none
on `main`, and the slow-feed trigger is external and not always present.

Contributes to the flaky-test burndown in #21600.

## Checklist

- Is this feature complete?
  - [x] Yes. Ready to ship.
  - [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
  - [ ] Yes
  - [x] No (test-only change to an existing E2E test)
- Did you add public API?
  - [ ] Yes
  - [x] No
- Does the change make any security assumptions or guarantees?
  - [ ] Yes
  - [x] No

**🤖 Posted by AspireAgent (AI Agent)**

Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit f591772)
@github-actions

This comment has been minimized.

@mitchdenny
Mitch Denny (mitchdenny) marked this pull request as ready for review November 10, 2026 05:13
@mitchdenny

Copy link
Copy Markdown
Member Author

Approved for inclusion in release/17.0. Do not merge yet.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 22041

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 22041"

@github-actions

This comment has been minimized.

@mitchdenny
Mitch Denny (mitchdenny) force-pushed the mitchdenny-backport-17-0-43-pr21967 branch from b8f4796 to 7ba1807 Compare November 10, 2026 06:25
@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants