Skip to content

Build the CLI before MCP initialization in McpDocsE2ETests - #21450

Merged
Eric Erhardt (eerhardt) merged 2 commits into
mainfrom
agent/fix-mcpdocs-e2e-cli-build
Nov 5, 2026
Merged

Eric Erhardt (eerhardt) merged 2 commits into
mainfrom
agent/fix-mcpdocs-e2e-cli-build

Conversation

@davidfowl

@davidfowl David Fowler (davidfowl) commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Fixes the intermittent outerloop failure of Aspire.Cli.EndToEnd.Tests.McpDocsE2ETests.GetDoc_WithInvalidSlug_ReturnsError. It fails with System.TimeoutException: Initialization timed out from McpClient.CreateAsync in InitializeAsync.

Denominator

  • Window: scheduled Outerloop Tests (tests-outerloop.yml) on main, Oct 16–30 2026.
  • Runs: 15 scheduled runs (14 failed, 1 cancelled). In those 14 failed first attempts, this test failed in 3: runs 39710045918 (Oct 22), 40153523485 (Oct 25) and 40804532657 (Oct 29). The other failures in those runs are the persistent Templates Redis and Radius deploy breaks.
  • Intermittent: auto-rerun attempts of the same commit passed. In run 40153523485, attempt 1 failed while attempts 2–4 passed. In run 39888289397, attempt 1 failed and attempt 2 passed.
  • No existing issue or PR references this test.

Root cause

InitializeAsync started the server with dotnet run --project src/Aspire.Cli/Aspire.Cli.csproj -- agent mcp. The Cli.EndToEnd job installs the CLI from an archive and never builds src/Aspire.Cli from source; the test project doesn't reference it. So the first test in the class compiles the whole CLI before the server can answer the MCP initialize request, all inside McpClient's 60s initialization timeout.

TRX from 6 attempts across the two runs above:

  • The first test (always GetDoc_WithInvalidSlug_ReturnsError) took 59.5s–68.6s, right at the limit.
  • Every later test took 6.5s–12.3s.
  • Each failure was at exactly 65.1s, with VBCSCompiler at about 148% CPU in the job heartbeat at that moment.

Fix

  • Per review, the tests no longer build or dotnet run the CLI from source. A class fixture installs the CLI selected by CliInstallStrategy.Detect() into an isolated temp prefix using the existing get-aspire-cli*.sh acquisition scripts (with --skip-path --skip-completions --skip-extension), then each test launches that aspire executable with agent mcp.
  • On CI, ASPIRE_E2E_CLI_ARCHIVE_DIR selects LocalArchive, so the tests exercise the packaged native CLI built for the commit. PullRequest and InstallScript modes are supported the same way; host-mutating or Docker-only modes fail with a clear message.
  • The fixture checks aspire --version against the strategy's expected version, mirroring VerifyAspireCliVersionAsync. Install happens before McpClient.CreateAsync, so the 60s timeout covers only the MCP handshake.
  • No assertions, timeouts, retries or sleeps changed.

Validation (local, linux-x64)

Result
Before (main a76b57a9, cold CLI build state) ListTools_IncludesDocsTools failed in 1m 05.3s: System.TimeoutException : Initialization timed out (same signature as CI)
After, LocalArchive with this PR's cli-native-archives-linux-x64 CI artifact 11/11 passed; each test 5.1s–7.6s, including the first. Install took about 1.3s. src/Aspire.Cli was not built.
After, PullRequest mode (GITHUB_PR_NUMBER=21450) Passed
Version guard with a mismatched Aspire.Cli.*.nupkg Setup fails with Aspire CLI version mismatch as intended

InstallScript mode couldn't be exercised locally because aka.ms doesn't resolve in my environment.

Fixes: no tracking issue exists; found by scanning the scheduled outerloop runs.

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes (the existing E2E tests are the regression coverage; they fail before and pass after in the cold-CLI state)
    • No
  • Did you add public API?
    • Yes
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
    • No

🤖 Posted by AspireAgent (AI Agent)

The outerloop Cli.EndToEnd job installs the CLI from an archive and never
builds src/Aspire.Cli, so the first test's 'dotnet run --project' compiled
the CLI inside McpClient.CreateAsync's 60s initialization timeout. Build
the CLI once in a class fixture and launch it with --no-build.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21450

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21450"

@aspire-repo-bot
aspire-repo-bot Bot requested a balanced review from Copilot October 30, 2026 11:49
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are recorded for the change moving CLI compilation outside the MCP initialization timeout.

No open issues 🎉

What changed in this PR

Moves CLI compilation outside the MCP initialization timeout to address intermittent failures in the documentation end-to-end tests.

Changes:

  • Adds a class fixture that builds the CLI once and reports build output on failure.
  • Launches each MCP server with dotnet run --no-build, keeping compilation outside the handshake window.
  • Preserves existing assertions, timeouts, and retry behavior.
File Description
tests/​Aspire.Cli.EndToEnd.Tests/​McpDocsE2ETests.cs Adds the shared CLI build fixture and skips compilation during MCP startup.

Review effort: Balanced

Copy link
Copy Markdown
Collaborator Author

Exact-head outerloop validation (head 14083e0b)

Outerloop was dispatched on this branch: run 41050874298, Cli.EndToEnd job 124060452392. Results come from the TRX in the logs-Cli.EndToEnd-ubuntu-latest artifact.

  • McpDocsE2ETests: 11/11 passed. The first test (GetDoc_WithInvalidSlug_ReturnsError) took 10.8s; the other tests took 6.6–9.0s.
    • Before this change, the first test in scheduled runs took 59.5–68.6s, and the failures happened at the 60s MCP initialization timeout (3/14 scheduled outerloop runs, Oct 16–30).
  • The only failure in this job was RadiusDeployTests.DeployRadiusRedisRabbitMqAndMongoBackingResourcesToKind. It is unrelated: that test also failed in 14/14 scheduled runs over the same window.

Required PR CI at this head: all checks green.

🤖 Posted by AspireAgent (AI Agent)

Copy link
Copy Markdown
Collaborator Author

Follow-up on outerloop run 41050874298, latest attempt 4, head 14083e0b. None of the red outerloop jobs are caused by this PR, which only changes tests/Aspire.Cli.EndToEnd.Tests/McpDocsE2ETests.cs. Main's scheduled outerloop 41158473015 (Oct 31) fails the same jobs with the same signatures:

  • Cli.EndToEnd: only RadiusDeployTests.DeployRadiusRedisRabbitMqAndMongoBackingResourcesToKind fails ("detected ERR prompt at sequence 46", RadiusDeployTests.cs:711), the same as on main. Main's Oct 31 run also failed McpDocsE2ETests.GetDoc_WithInvalidSlug_ReturnsError, the test this PR fixes. That test passes on this branch.
  • Hosting.Testing: DashboardUrlTests.DashboardDisplaysResourceUpdatesWhileAppHostIsPausedAtBreakpoint fails with Playwright "Locator expected to be visible", the same as on main.
  • Templates and Final Results: red on main as well.

These are existing main breaks rather than flakes, so I haven't rerun anything. Required PR CI on this head is all green, with 0 unresolved threads.

🤖 Posted by AspireAgent (AI Agent)

Copy link
Copy Markdown
Collaborator Author

Outerloop triage for head 14083e0b, run 41050874298 (attempt 4). The 3 red outerloop jobs don't come from this PR. The PR only changes McpDocsE2ETests.cs, and none of the failing tests is in that file. All three failures reproduce on main's daily scheduled outerloop, and each one already has an owner:

Job Failing test Same failure on main Owner
Hosting.Testing (ubuntu) DashboardUrlTests.DashboardDisplaysResourceUpdatesWhileAppHostIsPausedAtBreakpoint (Playwright expect, DashboardUrlTests.cs:130) 11-01 run #21655 / #21656
Cli.EndToEnd (ubuntu) RadiusDeployTests.DeployRadiusRedisRabbitMqAndMongoBackingResourcesToKind (ERR prompt at sequence 46) 11-01, 10-30 #21414 / #21422
Templates (ubuntu) StarterTemplateWithRedisCacheTests_Net11 (session aborted, 3/3 failed) 11-01 run #21413 / #21423

Main's scheduled Outerloop run has been red on all 14 of the last 14 daily runs, and each of those already went to attempt 4. I didn't rerun these jobs because the failures are deterministic on main, so another attempt can't go green until the owning PRs merge. CI (ci.yml) passed on attempt 1 on this head.

🤖 Posted by AspireAgent (AI Agent)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found 1 test-coverage issue: the MCP docs E2E tests rebuild and run the CLI from source instead of exercising the prebuilt CLI artifact.

Replace the source-build fixture with one that installs the CLI selected by
CliInstallStrategy.Detect() into an isolated temp prefix using the existing
get-aspire-cli*.sh acquisition scripts, then launches that executable with
`agent mcp`. On CI, ASPIRE_E2E_CLI_ARCHIVE_DIR selects LocalArchive, so the
tests exercise the packaged native CLI for the commit and no longer compile
src/Aspire.Cli. The installed version is verified against the strategy's
expected version, mirroring VerifyAspireCliVersionAsync.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Tests selector

1 / 100 PR test projects · 0 PR jobs, from 1 changed file.

Selected PR test projects (1 / 100)

Aspire.Cli.EndToEnd.Tests

Selected PR jobs (0)

none


How these were chosen — grouped by what changed

🧪 tests/Aspire.Cli.EndToEnd.Tests/McpDocsE2ETests.cs (changed test)
→ 1 directly: Aspire.Cli.EndToEnd.Tests

Job reasons

none


Selection computed for commit 733499e.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the change to launch CLI documentation MCP tests with an installed CLI.

No open issues 🎉

Review effort: Balanced

@github-actions

github-actions Bot commented Nov 2, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@davidfowl David Fowler (davidfowl) added the agent:working An AI agent is actively working on this issue or PR label Nov 5, 2026 — with Copilot SWE Agent
@davidfowl David Fowler (davidfowl) removed the agent:working An AI agent is actively working on this issue or PR label Nov 5, 2026
@eerhardt
Eric Erhardt (eerhardt) merged commit ac6bf3b into main Nov 5, 2026
315 of 318 checks passed
@microsoft-github-policy-service microsoft-github-policy-service Bot added this to the 17.0 milestone Nov 5, 2026
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

✅ No documentation update needed.

docs_optional → test_only. Triggered signal: pr_body_has_cli_flag_mention (a false positive: the PR body only mentions a dotnet run ... -- agent mcp dev command). The only changed file is tests/Aspire.Cli.EndToEnd.Tests/McpDocsE2ETests.cs, which matches tests/** and has no user-facing change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants