Skip to content

Bound GitRepository's post-exit output drain so git commands can't hang - #21562

Open
David Fowler (davidfowl) wants to merge 5 commits into
microsoft:mainfrom
davidfowl:fix-21141-git-ls-files-drain
Open

David Fowler (davidfowl) wants to merge 5 commits into
microsoft:mainfrom
davidfowl:fix-21141-git-ls-files-drain

Conversation

@davidfowl

@davidfowl David Fowler (davidfowl) commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Description

GitRepositoryTests time out after 30s on macOS (#21141, #20729). In every failing log I checked, Aspire.Cli.Git.GitRepository logs Listing git-included files under: … and then never logs git ls-files returned ….

Root cause

#17079 moved GitRepository to Process.RunAndCaptureTextAsync. That API waits for EOF on stdout and stderr before it waits for the process to exit, and it has no bound. EOF only arrives when every copy of the pipe write ends is closed. That doesn't happen if another process holds an inherited copy. Two ways that can happen:

  • a hook or helper that git spawns and that outlives git, or
  • a process spawned concurrently that inherits the handles. On macOS the runtime creates pipes with pipe() followed by a separate fcntl(FD_CLOEXEC), which leaves a window where this can happen.

In either case GetIncludedFilesAsync and GetRootAsync hang forever even though git has already exited. The repo documents the same hazard in ProcessCaptureRunner, which already bounds its post-exit drain for this reason. Before #17079, GitRepository read output concurrently with WaitForExitAsync instead of requiring EOF first.

All occurrences I found are after #17079 merged (2026-10-01).

Fix

GitRepository.RunGitAsync now:

  • waits for git to exit, then bounds the output drain to 2s (the same bound ProcessCaptureRunner uses). In the normal case git closes its pipes on exit and the drain completes immediately.
  • uses the complete stdout if only stderr is held open.
  • returns null if stdout itself is held open, because the output may be truncated. null already means "git acceleration unavailable", so callers fall back to the filesystem walker.
  • still kills git on cancellation, as before.
  • reads each pipe's raw bytes and decodes them with Encoding.GetString, like Process.RunAndCaptureTextAsync. This keeps a leading U+FEFF in the first path; StreamReader.ReadToEndAsync would strip it.

There are no retries, no sleeps and no test timeout changes.

I didn't reuse ProcessCaptureRunner. On drain expiry it returns an empty capture with exit code 0, which ls-files would read as "zero files", and #21556 is currently changing that code.

Regression test

GetIncludedFilesAsync_DescendantHoldsInheritedPipe_ReturnsAfterGitExits (POSIX only) sets core.fsmonitor to a hook script. git runs the hook during ls-files, and the hook leaves sleep 300 >/dev/null & behind, which holds git's stderr pipe after git exits. The test kills the leftover processes in finally.

GetIncludedFilesAsync_DescendantHoldsStdoutAfterGitExits_ReturnsNull (POSIX only) covers the case where stdout is held. It uses a fake git, through a new optional internal gitExecutable constructor parameter that defaults to "git". The fake git writes partial output, leaves a process holding stdout and exits 0. The test asserts that the method returns null (fall back to the walker) instead of hanging or returning a partial list.

GetIncludedFilesAsync_FirstPathStartsWithByteOrderMark_PreservesPath (POSIX only) checks that decoding still matches the previous path. Before the fix it fails because the first path comes back with the U+FEFF stripped.

Before the fix (local DefaultTimeout is 5s; 30s in CI). The failure has the same log signature as CI:

failed GitRepositoryTests.GetIncludedFilesAsync_DescendantHoldsInheritedPipe_ReturnsAfterGitExits (5s 048ms)
  System.TimeoutException : The operation at .../GitRepositoryTests.cs:184 timed out after reaching the limit of 5000
  [09:09:13] GitRepository Debug: Listing git-included files under: /tmp/Aspire.Cli.Tests/Workspace/znlkum2r.2cs
  [09:09:18] GitRepository Debug: git ls-files returned 2 files ...   <- only after the test's finally killed the leftover process

Both new tests time out like this on the old code (5s TimeoutException).

After the fix:

  • Both new tests pass. The stderr test passes in about 3s and returns git's file list.
  • GitRepositoryTests, GitCommandRunnerTests, GitClientTests and AppHostCandidateFinderTests: 94 passed, 0 failed.
  • A bounded loop of 5 runs of GitRepositoryTests passed 5/5.
  • No leftover sleep processes after the runs.

Related

These show the same Listing git-included files signature with no returned line, through ambient AppHost discovery. They aren't closed here, so they can be confirmed separately: #20940, #20911, #20737 (and likely #20939).

Fixes #21141
Fixes #20729

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
    • No

🤖 Posted by AspireAgent (AI Agent)

GitRepository ran git through Process.RunAndCaptureTextAsync, which waits for EOF
on stdout and stderr with no bound before it waits for exit. If any other process
holds an inherited copy of git's pipe write ends (a hook or helper git spawned, or
a process spawned concurrently that inherited the handles), EOF never arrives and
GetRootAsync/GetIncludedFilesAsync hang even though git has exited.

Wait for git to exit, then bound the output drain to 2s. If only stderr is held
open, the complete stdout is still used; if stdout is held open, return null so
callers fall back to the filesystem walker instead of using a partial list.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21562

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21562"

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The regression test covers only stderr remaining open, leaving the bounded null-return fallback for open stdout unverified and partial results potentially able to silently omit AppHosts.

1 open issue
What changed in this PR

This PR prevents CLI Git discovery from hanging when inherited output pipes remain open after Git exits, addressing the macOS timeouts reported in #21141 and #20729.

Changes:

  • Bounds post-exit output draining to two seconds while preserving cancellation handling.
  • Keeps complete stdout when only stderr remains open; returns null for incomplete stdout so discovery can use its filesystem fallback.
  • Adds a POSIX regression test with a descendant holding stderr open and cleanup of leftover processes.
File Description
tests/​Aspire.Cli.Tests/​Git/​GitRepositoryTests.cs Adds inherited-pipe regression coverage and process cleanup.
src/​Aspire.Cli/​Git/​GitRepository.cs Bounds output draining and handles incomplete output safely.

Review effort: Balanced

Comment thread src/Aspire.Cli/Git/GitRepository.cs Resolved
Add an internal git executable override so a test can run a fake git that exits
while a descendant still holds its stdout pipe, and assert GetIncludedFilesAsync
returns null within the bounded wait instead of hanging or returning a partial list.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The redirected output and error readers are not explicitly disposed, leaving two pipe handles per invocation awaiting finalization.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Low severity Dispose redirected output readers after process execution

src/​Aspire.Cli/​Git/​GitRepository.cs:161

Please dispose both redirected readers explicitly: Process.Dispose() does not close externally accessed StandardOutput and StandardError readers, and cancelling readCts only cancels their reads. Each invocation therefore leaves two pipe handles awaiting finalization, even on success. Scope both readers with using so they are closed after the finally block cancels pending reads.

1 resolved since last review

Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Process.Dispose doesn't close redirected streams once they've been accessed, so
dispose the readers explicitly to release the pipe handles promptly.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator Author

CI and review update:

🤖 Posted by AspireAgent (AI Agent)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

StandardOutput strips a leading U+FEFF from the first Git-listed filename, producing an incorrect authoritative path that breaks AppHost discovery and needs BOM-preserving decoding with a regression test.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Preserve leading BOM in Git-emitted paths

src/​Aspire.Cli/​Git/​GitRepository.cs:161

Process.StandardOutput reintroduces BOM stripping, so if the first path emitted by git ls-files -z starts with U+FEFF (for example, \uFEFFsrc/AppHost.csproj), that character is lost. The returned set then contains the wrong path, and AppHost discovery treats it as authoritative rather than falling back to the filesystem. Decode stdout without consuming a BOM or encoding preamble, and add a regression test that preserves this filename.

Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Closing a synchronous pipe handle on Windows can wait for a read that's still
pending on another thread, which is exactly the case when another process holds
git's pipe open. Dispose each reader from a continuation on its read instead, so
the handles are released promptly without ever blocking the caller.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator Author

CI update for bd3018e (run 41192229293):

  • Cli.EndToEnd-StopNonInteractiveTests.StopAllAppHostsFromAppHostDirectory failed on attempt 1. The AppHost couldn't bind a fixed port: Failed to bind to address https://127.0.0.1:22001: address already in use. The github-actions bot reran it in attempt 2 and it passed.
  • VS Code extension E2E (Windows): command-palette and settings-files. Both attempt 1 jobs ended at the same moment, 47 minutes after starting, with no failed step and no log blob, which is the runner-loss signature. In attempt 2, command-palette passed. settings-files ended the same way, again after 47 minutes with no failed step and no log. Another run in the same window (41194399981) also had a Windows VS Code E2E job (debug-startup-timeout) hang for 43 minutes. On 14b13f5, the previous head of this PR, settings-files passed in about 4 minutes.

I couldn't completely rule this PR out, so I re-checked the reader disposal added in bd3018e and found a real Windows hazard. Process uses synchronous pipe handles on Windows. Closing a handle while another thread is blocked in ReadFile on it can wait for that read to finish, which is exactly what happens when another process holds git's pipe open. 898f403 now disposes each reader from a continuation on its read, so handles are still released promptly (immediately in the normal case) and disposal can never block the caller. Locally, the Git and AppHostCandidateFinder tests pass (94 passed, 0 failed). New CI is running on 898f403.

🤖 Posted by AspireAgent (AI Agent)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because a subtle stdout-decoding change can strip a leading U+FEFF from filenames and silently omit an AppHost from discovery.

No open issues 🎉

Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

StreamReader.ReadToEndAsync strips a leading byte order mark, but the
previous Process.RunAndCaptureTextAsync path decoded raw bytes. Read the
raw bytes and decode them with the same encoding so the first path from
git ls-files -z isn't changed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator Author

Addressed the Copilot overview note on 898f403 in d728d53.

StreamReader.ReadToEndAsync strips a leading byte order mark, so a first git ls-files -z entry starting with U+FEFF would have lost that character. The previous Process.RunAndCaptureTextAsync path decoded the raw bytes with Encoding.GetString, which keeps it. RunGitAsync now copies the raw bytes from each pipe's BaseStream and decodes them with the same encoding, so behaviour matches the old path.

New test GetIncludedFilesAsync_FirstPathStartsWithByteOrderMark_PreservesPath:

  • Before: fails, because the first path comes back as AppHost.csproj with the U+FEFF stripped.
  • After: passes. GitRepositoryTests and AppHostCandidateFinderTests: 33 passed, 0 failed, 1 skipped.

CI on 898f403: 194 checks passed. The Windows VS Code command-palette and settings-files jobs passed in about 4 minutes each, so the 47-minute hangs from bd3018e didn't recur. The only failure was VS Code extension E2E (Linux, edge-cases): three tests timed out after 180s waiting for AppHost resources or debug startup. The bot retried that job; d728d53 supersedes it.

🤖 Posted by AspireAgent (AI Agent)

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the change to prevent git-based discovery from hanging on inherited output pipes.

No open issues 🎉

Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Copy link
Copy Markdown
Collaborator Author

CI on d728d53, run 41204248633, attempt 1: 192 passed and 3 failed. All three are known infrastructure flakes. The bot's automatic attempt 2 counts as the single rerun.

  • VS Code extension E2E (Windows, tree-actions): lost after 47m and VS Code extension E2E (Windows, debug-dashboard): lost after 49m. Both runners died partway through "Run extension E2E tests": no step reached a conclusion, and the log endpoint returns BlobNotFound. These jobs normally take 6–10 minutes (on 898f403 they finished in 6m and 10m). Main push run 41175594640 on 6a8f531 has the same 47-minute Windows debug-dashboard failure without this change, so this is the Windows runner loss / VS Code Windows hang flake ([CI Failure] Flaky: VS Code extension E2E (Windows, tree-actions) 'select and execute run and debug pipeline steps' times out waiting for picker prompt #21180).
  • Aspire CLI Starter Validation (Ubuntu ARM64): the CLI found the AppHost and aspire start returned 0. After __dotnet-project-build finished, apiservice and webfrontend both went Starting -> FailedToStart, so aspire wait exited with 18. The other starter legs passed. This is the known CLI starter flake. It's in resource launch, not in the git file listing this PR changes.

🤖 Posted by AspireAgent (AI Agent)

David Fowler (davidfowl) commented Nov 11, 2026 •

Copy link
Copy Markdown
Collaborator Author

AspireAgent stopped: the conflict with main is resolved locally, but pushing the merge is blocked because main's incoming changes touch .github/workflows and this token lacks the workflows permission.

🤖 Posted by AspireAgent (AI Agent)

@davidfowl David Fowler (davidfowl) removed the agent:working An AI agent is actively working on this issue or PR label Nov 11, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The bounded post-exit drain prevents inherited Git pipes from hanging discovery while preserving complete stdout and falling back safely for incomplete output. The change is limited to internal CLI process handling, adds no public API, and preserves existing discovery behavior on normal and failure paths.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-cli ci-health Fixes CI flakiness or CI infrastructure (AspireAgent queue)

Projects

None yet

3 participants