Skip to content

[release/17.0] Fix RepositoryTemplate_ResolvesRefsAndReusesPinnedCheckoutsAcrossInvocations timeout on slow Git checkouts - #22039

Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
mitchdenny-backport-17-0-40-pr21955from
mitchdenny-backport-17-0-41-pr21952
Closed

Mitch Denny (mitchdenny) wants to merge 1 commit into
mitchdenny-backport-17-0-40-pr21955from
mitchdenny-backport-17-0-41-pr21952

Conversation

@mitchdenny

Copy link
Copy Markdown
Member

Description

Backport of #21952 to release/17.0 (cherry-picked from 4563cf8 with no modifications).

Layer 41 of 55 in the release/17.0 backport stack; stacked on #22038.

Backport recommendation: No — Test flake.

Validation: Stable patch IDs match. Builds/tests not run for this unchanged cherry-pick.

Backport of #21952

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

…cations timeout on slow Git checkouts (#21952)

**What failed.**
`NewCommandLocalTemplateTests.RepositoryTemplate_ResolvesRefsAndReusesPinnedCheckoutsAcrossInvocations`
timed out on `main`: `System.TimeoutException : The operation at
…NewCommandLocalTemplateTests.cs:1700 timed out after reaching the limit
of 30000ms` (Windows:
https://github.com/microsoft/aspire/actions/runs/41534093007; macOS:
https://github.com/microsoft/aspire/actions/runs/40534267321).

**Why.** Each `aspire new --template-repository` call in this test can
do a fresh checkout with real Git: `init`, then `fetch --depth 1`,
`checkout` and `rev-parse`. The test wraps every call in
`DefaultTimeout()`, which is 30s on CI. The product gives each Git phase
two minutes (`TemplateEngineFactory.s_gitOperationTimeout`). The failing
Windows run logged checkouts of 27.5s (`default`) and 27.0s
(`branch-with-slash`). Then `moved-branch` spent 8.3s in `git init` and
was still in `fetch` after 23.7s, so it crossed 30s while the product's
own deadline was nowhere near expiring.

**Fix.** The test's guard for each invocation now uses
`TestConstants.LongTimeoutTimeSpan` (180s on CI), which sits above the
product's per-phase Git budget. All assertions are unchanged, and no
sleeps or retries were added. A Git hang is still caught: first by the
product's 2-minute deadline, then by the 180s test guard.

Fixes #21704

## Confidence

The Windows signature is reproduced exactly. A `git` shim on `PATH` adds
the latency seen in the failing run (8s to `init`, 23s to `fetch`), only
for product invocations (`--literal-pathspecs`). It runs with
`GITHUB_JOB` set so the CI timeouts apply. Before the fix, every run
fails with the same `cs:1700 … 30000ms` timeout. After the fix, every
run passes in about 95s. Without the shim, the full
`NewCommandLocalTemplateTests` class passes (529/529). Main uncertainty:
the latency is emulated on Linux rather than measured on a Windows
runner. The October macOS hits came alongside other 30s timeouts in the
same job (#21335), so part of that cluster was runner
stalls. The fix tolerates those up to 180s per invocation, but it can't
help a stall longer than that.

**How we stop this recurring:** tests that run real Git acquisition
should set their hang guard from the product's acquisition budget, not
the generic 30s `DefaultTimeout`.

## Failure stats (14d, attempt 1)

- **Test / signature:**
`Aspire.Cli.Tests.Commands.NewCommandLocalTemplateTests.RepositoryTemplate_ResolvesRefsAndReusesPinnedCheckoutsAcrossInvocations`,
`System.TimeoutException … timed out after reaching the limit of
30000ms` in the per-invocation `InvokeAsync().DefaultTimeout()`.
- **Failures / runs:** 13/1175 (1.1%). `main` 4/160 (2.5%); PRs 9/1015
(0.9%). One PR-caused hit is excluded:
https://github.com/microsoft/aspire/actions/runs/42127224939, a snapshot
change on the provenance branch that failed in 9s with no timeout. 18 of
258 failed Cli job logs could not be downloaded (404), so the tracker
#21600 counts 15.
- **Rerun-pass:** 6 PR runs passed on a later attempt.
- **Last hit:** 2026-11-02 on `main` (Windows). Up to 5 runs:
https://github.com/microsoft/aspire/actions/runs/41534093007,
https://github.com/microsoft/aspire/actions/runs/40601737425,
https://github.com/microsoft/aspire/actions/runs/40534267321,
https://github.com/microsoft/aspire/actions/runs/40498007795,
https://github.com/microsoft/aspire/actions/runs/40421037456
- **Still hitting main:** yes. The last `main` hit is 2026-11-02
(Windows), after the macOS cluster of 10-26 to 10-27.
- **Local N/N:** before 5/5 failed, after 5/5 passed (Git latency
emulated as above).
- **This PR's CI:** green at cfd1146.
`Aspire.Cli.Tests` passed on Windows, macOS and Ubuntu
(https://github.com/microsoft/aspire/actions/runs/42569185627). In
attempt 1, only "Lint handwritten workflows" failed, on a 504 while
downloading actionlint, and it passed on the automatic rerun.
- **Verdict:** MODERATE. The Windows root cause is shown in the CI log
and reproduced with exact timings. The fix stays in the test layer and
matches the product's contract. The occurrence rate is low and part of
the history comes from macOS runner stalls.

**🤖 Posted by AspireAgent (AI Agent)**

Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit 4563cf8)
@github-actions

This comment has been minimized.

@mitchdenny
Mitch Denny (mitchdenny) force-pushed the mitchdenny-backport-17-0-41-pr21952 branch from 6488372 to fda9c74 Compare November 10, 2026 04:23
@mitchdenny
Mitch Denny (mitchdenny) force-pushed the mitchdenny-backport-17-0-41-pr21952 branch from fda9c74 to 6488372 Compare November 10, 2026 04:25
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 22039

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 22039"

@mitchdenny
Mitch Denny (mitchdenny) marked this pull request as ready for review November 10, 2026 05:10
@mitchdenny

Copy link
Copy Markdown
Member Author

Approved for inclusion in release/17.0. Do not merge yet.

@github-actions

This comment has been minimized.

@mitchdenny
Mitch Denny (mitchdenny) force-pushed the mitchdenny-backport-17-0-41-pr21952 branch from 0200e37 to 6488372 Compare November 10, 2026 06:28
@mitchdenny
Mitch Denny (mitchdenny) force-pushed the mitchdenny-backport-17-0-41-pr21952 branch from 6488372 to 0200e37 Compare November 10, 2026 07:53
@mitchdenny
Mitch Denny (mitchdenny) force-pushed the mitchdenny-backport-17-0-41-pr21952 branch from 0200e37 to 6488372 Compare November 10, 2026 08:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants