From 64883726f48633e10ada28bf127f327981dc1236 Mon Sep 17 00:00:00 2001 From: David Fowler Date: Mon, 9 Nov 2026 14:21:21 -0800 Subject: [PATCH] Fix RepositoryTemplate_ResolvesRefsAndReusesPinnedCheckoutsAcrossInvocations timeout on slow Git checkouts (#21952) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit **What failed.** `NewCommandLocalTemplateTests.RepositoryTemplate_ResolvesRefsAndReusesPinnedCheckoutsAcrossInvocations` timed out on `main`: `System.TimeoutException : The operation at …NewCommandLocalTemplateTests.cs:1700 timed out after reaching the limit of 30000ms` (Windows: https://github.com/microsoft/aspire/actions/runs/41534093007; macOS: https://github.com/microsoft/aspire/actions/runs/40534267321). **Why.** Each `aspire new --template-repository` call in this test can do a fresh checkout with real Git: `init`, then `fetch --depth 1`, `checkout` and `rev-parse`. The test wraps every call in `DefaultTimeout()`, which is 30s on CI. The product gives each Git phase two minutes (`TemplateEngineFactory.s_gitOperationTimeout`). The failing Windows run logged checkouts of 27.5s (`default`) and 27.0s (`branch-with-slash`). Then `moved-branch` spent 8.3s in `git init` and was still in `fetch` after 23.7s, so it crossed 30s while the product's own deadline was nowhere near expiring. **Fix.** The test's guard for each invocation now uses `TestConstants.LongTimeoutTimeSpan` (180s on CI), which sits above the product's per-phase Git budget. All assertions are unchanged, and no sleeps or retries were added. A Git hang is still caught: first by the product's 2-minute deadline, then by the 180s test guard. Fixes microsoft/aspire#21704 ## Confidence The Windows signature is reproduced exactly. A `git` shim on `PATH` adds the latency seen in the failing run (8s to `init`, 23s to `fetch`), only for product invocations (`--literal-pathspecs`). It runs with `GITHUB_JOB` set so the CI timeouts apply. Before the fix, every run fails with the same `cs:1700 … 30000ms` timeout. After the fix, every run passes in about 95s. Without the shim, the full `NewCommandLocalTemplateTests` class passes (529/529). Main uncertainty: the latency is emulated on Linux rather than measured on a Windows runner. The October macOS hits came alongside other 30s timeouts in the same job (microsoft/aspire#21335), so part of that cluster was runner stalls. The fix tolerates those up to 180s per invocation, but it can't help a stall longer than that. **How we stop this recurring:** tests that run real Git acquisition should set their hang guard from the product's acquisition budget, not the generic 30s `DefaultTimeout`. ## Failure stats (14d, attempt 1) - **Test / signature:** `Aspire.Cli.Tests.Commands.NewCommandLocalTemplateTests.RepositoryTemplate_ResolvesRefsAndReusesPinnedCheckoutsAcrossInvocations`, `System.TimeoutException … timed out after reaching the limit of 30000ms` in the per-invocation `InvokeAsync().DefaultTimeout()`. - **Failures / runs:** 13/1175 (1.1%). `main` 4/160 (2.5%); PRs 9/1015 (0.9%). One PR-caused hit is excluded: https://github.com/microsoft/aspire/actions/runs/42127224939, a snapshot change on the provenance branch that failed in 9s with no timeout. 18 of 258 failed Cli job logs could not be downloaded (404), so the tracker microsoft/aspire#21600 counts 15. - **Rerun-pass:** 6 PR runs passed on a later attempt. - **Last hit:** 2026-11-02 on `main` (Windows). Up to 5 runs: https://github.com/microsoft/aspire/actions/runs/41534093007, https://github.com/microsoft/aspire/actions/runs/40601737425, https://github.com/microsoft/aspire/actions/runs/40534267321, https://github.com/microsoft/aspire/actions/runs/40498007795, https://github.com/microsoft/aspire/actions/runs/40421037456 - **Still hitting main:** yes. The last `main` hit is 2026-11-02 (Windows), after the macOS cluster of 10-26 to 10-27. - **Local N/N:** before 5/5 failed, after 5/5 passed (Git latency emulated as above). - **This PR's CI:** green at cfd1146eb63e1e48b5408150bcf071fdd7c742e2. `Aspire.Cli.Tests` passed on Windows, macOS and Ubuntu (https://github.com/microsoft/aspire/actions/runs/42569185627). In attempt 1, only "Lint handwritten workflows" failed, on a 504 while downloading actionlint, and it passed on the automatic rerun. - **Verdict:** MODERATE. The Windows root cause is shown in the CI log and reproduced with exact timings. The fix stays in the test layer and matches the product's contract. The occurrence rate is low and part of the history comes from macOS runner stalls. **🤖 Posted by AspireAgent (AI Agent)** Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 4563cf8da832d2425db65ef130829336350cccff) --- .../Commands/NewCommandLocalTemplateTests.cs | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/Aspire.Cli.Tests/Commands/NewCommandLocalTemplateTests.cs b/tests/Aspire.Cli.Tests/Commands/NewCommandLocalTemplateTests.cs index d97bbace2ae..8b50b3cc833 100644 --- a/tests/Aspire.Cli.Tests/Commands/NewCommandLocalTemplateTests.cs +++ b/tests/Aspire.Cli.Tests/Commands/NewCommandLocalTemplateTests.cs @@ -1746,7 +1746,10 @@ async Task GenerateAsync(string label, string? reference) arguments.AddRange(["--template-repository-ref", reference]); } - var exitCode = await provider.GetRequiredService().Parse(arguments.ToArray()).InvokeAsync().DefaultTimeout(); + // Fresh checkouts run real git init/fetch/checkout, which take 23-27s on loaded Windows + // runners. The product bounds each Git phase at two minutes, so a 30s hang guard is too tight. + var exitCode = await provider.GetRequiredService().Parse(arguments.ToArray()).InvokeAsync() + .DefaultTimeout(TestConstants.LongTimeoutTimeSpan); outputHelper.WriteLine($"[{DateTimeOffset.UtcNow:O}] Generation '{label}' returned {exitCode} after {stopwatch.Elapsed}."); Assert.Equal(CliExitCodes.Success, exitCode);