Repository navigation
[release/17.0] Fix RepositoryTemplate_ResolvesRefsAndReusesPinnedCheckoutsAcrossInvocations timeout on slow Git checkouts - #22039
Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
Conversation
…cations timeout on slow Git checkouts (#21952) **What failed.** `NewCommandLocalTemplateTests.RepositoryTemplate_ResolvesRefsAndReusesPinnedCheckoutsAcrossInvocations` timed out on `main`: `System.TimeoutException : The operation at …NewCommandLocalTemplateTests.cs:1700 timed out after reaching the limit of 30000ms` (Windows: https://github.com/microsoft/aspire/actions/runs/41534093007; macOS: https://github.com/microsoft/aspire/actions/runs/40534267321). **Why.** Each `aspire new --template-repository` call in this test can do a fresh checkout with real Git: `init`, then `fetch --depth 1`, `checkout` and `rev-parse`. The test wraps every call in `DefaultTimeout()`, which is 30s on CI. The product gives each Git phase two minutes (`TemplateEngineFactory.s_gitOperationTimeout`). The failing Windows run logged checkouts of 27.5s (`default`) and 27.0s (`branch-with-slash`). Then `moved-branch` spent 8.3s in `git init` and was still in `fetch` after 23.7s, so it crossed 30s while the product's own deadline was nowhere near expiring. **Fix.** The test's guard for each invocation now uses `TestConstants.LongTimeoutTimeSpan` (180s on CI), which sits above the product's per-phase Git budget. All assertions are unchanged, and no sleeps or retries were added. A Git hang is still caught: first by the product's 2-minute deadline, then by the 180s test guard. Fixes #21704 ## Confidence The Windows signature is reproduced exactly. A `git` shim on `PATH` adds the latency seen in the failing run (8s to `init`, 23s to `fetch`), only for product invocations (`--literal-pathspecs`). It runs with `GITHUB_JOB` set so the CI timeouts apply. Before the fix, every run fails with the same `cs:1700 … 30000ms` timeout. After the fix, every run passes in about 95s. Without the shim, the full `NewCommandLocalTemplateTests` class passes (529/529). Main uncertainty: the latency is emulated on Linux rather than measured on a Windows runner. The October macOS hits came alongside other 30s timeouts in the same job (#21335), so part of that cluster was runner stalls. The fix tolerates those up to 180s per invocation, but it can't help a stall longer than that. **How we stop this recurring:** tests that run real Git acquisition should set their hang guard from the product's acquisition budget, not the generic 30s `DefaultTimeout`. ## Failure stats (14d, attempt 1) - **Test / signature:** `Aspire.Cli.Tests.Commands.NewCommandLocalTemplateTests.RepositoryTemplate_ResolvesRefsAndReusesPinnedCheckoutsAcrossInvocations`, `System.TimeoutException … timed out after reaching the limit of 30000ms` in the per-invocation `InvokeAsync().DefaultTimeout()`. - **Failures / runs:** 13/1175 (1.1%). `main` 4/160 (2.5%); PRs 9/1015 (0.9%). One PR-caused hit is excluded: https://github.com/microsoft/aspire/actions/runs/42127224939, a snapshot change on the provenance branch that failed in 9s with no timeout. 18 of 258 failed Cli job logs could not be downloaded (404), so the tracker #21600 counts 15. - **Rerun-pass:** 6 PR runs passed on a later attempt. - **Last hit:** 2026-11-02 on `main` (Windows). Up to 5 runs: https://github.com/microsoft/aspire/actions/runs/41534093007, https://github.com/microsoft/aspire/actions/runs/40601737425, https://github.com/microsoft/aspire/actions/runs/40534267321, https://github.com/microsoft/aspire/actions/runs/40498007795, https://github.com/microsoft/aspire/actions/runs/40421037456 - **Still hitting main:** yes. The last `main` hit is 2026-11-02 (Windows), after the macOS cluster of 10-26 to 10-27. - **Local N/N:** before 5/5 failed, after 5/5 passed (Git latency emulated as above). - **This PR's CI:** green at cfd1146. `Aspire.Cli.Tests` passed on Windows, macOS and Ubuntu (https://github.com/microsoft/aspire/actions/runs/42569185627). In attempt 1, only "Lint handwritten workflows" failed, on a 504 while downloading actionlint, and it passed on the automatic rerun. - **Verdict:** MODERATE. The Windows root cause is shown in the CI log and reproduced with exact timings. The fix stays in the test layer and matches the product's contract. The occurrence rate is low and part of the history comes from macOS runner stalls. **🤖 Posted by AspireAgent (AI Agent)** Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 4563cf8)
Closed
14 tasks
This comment has been minimized.
This comment has been minimized.
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-41-pr21952
branch
from
November 10, 2026 04:23
6488372 to
fda9c74
Compare
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-41-pr21952
branch
from
November 10, 2026 04:25
fda9c74 to
6488372
Compare
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 22039Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 22039" |
Mitch Denny (mitchdenny)
marked this pull request as ready for review
November 10, 2026 05:10
Member
Author
|
Approved for inclusion in release/17.0. Do not merge yet. |
Mitch Denny (mitchdenny)
requested review from
James Newton-King (JamesNK),
Adam Ratzman (adamint),
David Negstad (danegsta),
David Fowler (davidfowl),
Eric Erhardt (eerhardt),
Ella Hathaway (ellahathaway),
Karol Zadora-Przylecki (karolz-ms),
Ankit Jain (radical) and
Sébastien Ros (sebastienros)
as code owners
November 10, 2026 06:07
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-41-pr21952
branch
from
November 10, 2026 06:08
6488372 to
0200e37
Compare
This comment has been minimized.
This comment has been minimized.
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-41-pr21952
branch
from
November 10, 2026 06:28
0200e37 to
6488372
Compare
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-41-pr21952
branch
from
November 10, 2026 07:53
6488372 to
0200e37
Compare
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-41-pr21952
branch
from
November 10, 2026 08:38
0200e37 to
6488372
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #21952 to
release/17.0(cherry-picked from 4563cf8 with no modifications).Layer 41 of 55 in the release/17.0 backport stack; stacked on #22038.
Backport recommendation: No — Test flake.
Validation: Stable patch IDs match. Builds/tests not run for this unchanged cherry-pick.
Backport of #21952
Checklist
<remarks />and<code />elements on your triple slash comments?