Skip to content

Keep captured output when ProcessCaptureRunner's post-exit drain bound expires - #21556

Merged
Ankit Jain (radical) merged 3 commits into
mainfrom
aspireagent/fix-peer-probe-post-exit-drain
Nov 9, 2026
Merged

Ankit Jain (radical) merged 3 commits into
mainfrom
aspireagent/fix-peer-probe-post-exit-drain

Conversation

@davidfowl

@davidfowl David Fowler (davidfowl) commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Description

PeerInstallProbeTests.ProbeAsync_PeerEmitsArrayWithNonObjectFirstElement_FallsBackToVersion("[\"string\"]") failed once on Windows with Peer produced no usable output (and --version fallback). That run was 35003412018 / job 104497909578. The test took 4.3s. Passing runs take a median of 0.154s and a max of 0.51s (n=60, from TRX of 15 recent main Cli-windows runs, 300/300 PeerInstallProbe passes).

Root cause

ProcessCaptureRunner.RunAsync handles a process that exits normally like this:

  • It waits up to a fixed 2s (s_postExitCaptureWaitBound) for the stdout/stderr capture to reach EOF.
  • If that bound expires, SwallowCaptureAsync returns createEmptyCapture(). That throws away everything the reader had already buffered. Only after that does it cancel the reads.

So a peer that printed 9.0.0 and exited 0 could come back as exit 0 with empty stdout. ExtractVersionLine then returns null and the probe reports "no usable output". EOF arrives late when a descendant still holds the inherited pipe handle, or when reader continuations are delayed under CI load. The extra 2s matches the 4.3s failure duration.

The probe's Debug diagnostics, such as "Timed out waiting 2s for process output capture", were missing from the CI log. The test's LoggerFactory used the default Information minimum level, so they were filtered out before reaching the xunit provider.

Fix

When the post-exit drain bound expires, RunAsync now cancels the reads and then collects the capture, waiting at most the 250ms post-kill bound. It no longer abandons the capture. The result carries the real exit code, the capture, and a new ProcessCaptureFailureKind.OutputDrainTimedOut:

  • PeerInstallProbe treats OutputDrainTimedOut as success. ReadCappedAsync returns partial output on cancellation, so the version the peer wrote before exiting is kept. The existing parsers still reject unusable output.
  • GitCommandRunner and ConfigurationHelper reject any non-null FailureKind, so an incomplete drain still fails as before (RepositoryUnavailable / settings write refused). This keeps ConfigSourceServiceGitTests.RunProcess_DescendantHoldingOutputCannotBlockShutdown passing. The first revision, without the flag, broke that test in CI, which is why the flag exists.
  • The happy path doesn't change. The final timeoutCts.Cancel() still guarantees no capture task outlives the call.

I also changed the test logger factory to SetMinimumLevel(LogLevel.Trace) so future failures include the probe's Debug logs.

Regression test and evidence

The new test is ProbeAsync_VersionFallbackDescendantHoldsStdoutOpen_KeepsCapturedVersion (POSIX only). The fake peer:

  • fails doctor;
  • on --version, prints 9.0.0, leaves a background sleep 60 & that inherits stdout, and exits 0.

This makes the late-EOF condition deterministic. The test runs real processes through the real ProcessCaptureRunner; nothing is mocked.

Result
Before fix (Linux) Fails after 2.1s. The log shows Timed out waiting 2s for process output capture and the probe returns no version.
After fix (Linux) 10/10 pass, about 2.08s each (the expected drain bound, then cancellation).
PeerInstallProbeTests, ConfigSourceService*, GitCommandRunner*, ConfigurationHelper* and ProcessCaptureRunner* classes, after fix 340 total: 339 passed, 1 skipped (Windows-only), 0 failed. These run real git and real descendant processes.
New test plus RunProcess_DescendantHoldingOutputCannotBlockShutdown, repeated 5 × 5 = 25/25 passed

Caveat: the Windows failure left no probe diagnostics, so I can't prove the late EOF from that one log. One alternative remains possible: the fake peer's echo %* | findstr failing transiently, which exits 127 and produces the same message. The 2s-shaped extra duration points to the drain path, and the fix closes that path either way.

Fixes #20147

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
    • No
  • Does the change require an update in our Aspire docs?
    • Yes
    • No

🤖 Posted by AspireAgent (AI Agent)

ProcessCaptureRunner waited a fixed 2s for the output capture after the
process exited normally and, on timeout, discarded everything captured so
far and returned an empty capture. A peer that printed its version and
exited 0 could then be reported as "Peer produced no usable output" when
stdout EOF arrived late (a descendant holding the inherited pipe, or
delayed reader continuations under CI load).

When the drain bound expires, cancel the reads instead and collect the
capture, so captures that return partial output on cancellation (the
PeerInstallProbe readers) keep what was already written. Callers whose
captures throw on cancellation still get an empty capture as before.

Also let the probe tests' logger emit Debug diagnostics, which the factory's
default minimum level was filtering out.

Fixes #20147

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21556

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21556"

@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Post-cancellation output collection changes Git drain-timeout failures into success for exit-zero processes, breaking both parent-exit regression test cases.

1 open issue
What changed in this PR

Preserves buffered process output when the post-exit drain limit expires, addressing a possible cause of the peer-version probe failure tracked in #20147.

Changes:

  • Cancels pending reads before collecting partial output within a short additional bound.
  • Adds a POSIX regression test where a descendant keeps stdout open.
  • Enables Trace-level test logging so probe diagnostics appear in failure output.
File Description
tests/​Aspire.Cli.Tests/​Acquisition/​PeerInstallProbeTests.cs Adds inherited-pipe regression coverage and enables detailed logging.
src/​Aspire.Cli/​Utils/​ProcessCaptureRunner.cs Collects capture results after cancelling reads when the drain limit expires.

Review effort: Balanced

Callers such as GitCommandRunner must not treat output cut short by the
post-exit drain bound as complete. Report it as
ProcessCaptureFailureKind.OutputDrainTimedOut with the real exit code and
the partial capture; PeerInstallProbe accepts it because its parsers already
reject unusable output, while Git and ConfigurationHelper keep failing as
before.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the change preserving buffered process output when the post-exit drain limit expires.

1 open issue

Review effort: Balanced

Copy link
Copy Markdown
Collaborator Author

CI note for head f987c01 (run 41186503338): I reran the failed jobs once. Both failures are in the known VS Code extension Windows E2E family (#21180/#20063, Deno Windows fixture):

  • VS Code extension E2E (Windows, launch-profiles): the error was Timed out after 300000ms waiting for debug AppHost startup (DenoAppHost), followed by EBUSY ... rmdir ...\DenoAppHost during teardown. This job passed in each of the last 5 main runs I checked.
  • VS Code extension E2E (Windows, tree-actions): hung for more than 30 minutes; it normally takes about 6. I cancelled the run so the failed jobs could be rerun.

Neither is related to this change. Git and ConfigurationHelper still treat an incomplete post-exit drain as a failure; only the peer probe accepts it. The Cli unit tests on ubuntu, macOS and Windows and the CLI archive E2E matrix all passed on this head.

🤖 Posted by AspireAgent (AI Agent)

@davidfowl
David Fowler (davidfowl) marked this pull request as ready for review October 31, 2026 10:05
David Fowler (davidfowl) added a commit that referenced this pull request Nov 3, 2026
…gged after exit (#21626)

## Description


`ConfigSourceServiceGitTests.AcquireAndCheck_MatchRepositoryObjectFormat`
fails on Windows Cli jobs with `The service did not persist the Git
version check.` Over 7 days of ci.yml attempt-1 jobs it failed in 2 of
307 Windows PR jobs, 0 of 56 Windows main jobs, and 0 Linux/macOS jobs.
Details are in #21625.

### Root cause

In both failing logs, the check-phase `git ls-remote` logs `Git process
completed after 00:00:03.58; exit code 0` and no `output drained` line.
Other git calls in the same test take 0.05–0.27s.

`ProcessCaptureRunner` waits at most 2s (`captureTask.WaitAsync(2s)`)
for the capture after the process exits, so descendants holding
inherited pipes can't block callers. The bound's timer fires without a
thread-pool thread, but the stdout/stderr read continuations need one.
On Windows, `ReadAsync` on the synchronous redirected pipes goes through
the pool even for already-buffered data.

When the test host's pool is backlogged, the 2s bound expires while the
reads of an exited, fully written `git` are still queued. The capture is
discarded and `GitCommandRunner` maps it to `RepositoryUnavailable`. The
non-forced check swallows that, so `check.json` is never written.

### Fix

When the post-exit bound expires, `ProcessCaptureRunner` now measures
how long a newly queued global work item takes to run.
- If the pool runs it promptly (under 250ms), the reads aren't starved;
output really is being held open. The runner gives up exactly as before,
so the descendant-holding-pipe contract
(`RunProcess_DescendantHoldingOutputCannotBlockShutdown`) is unchanged.
- If the pool is backlogged, the runner keeps waiting for the capture
until the pool can schedule reads. Every post-exit wait, including the
probe, races capture completion against the operation token, so none of
them waits for the pool to recover. If the deadline or the caller's
cancellation fires before the output has drained, the result is
`TimedOut` or `Cancelled` like the pre-exit path, not a successful run
with partial output.

There are no sleeps or retries of the process, and no assertions were
weakened. The spec in `docs/specs/cli-config-sources.md` is updated to
match.

This is complementary to #21556. That PR keeps partial output (flagged
`OutputDrainTimedOut`) for `PeerInstallProbe`, but it intentionally
still fails Git on a drain timeout. This PR avoids that false drain
timeout when the output is actually complete.

### Evidence

- **Regression test**
`ProcessCaptureRunnerTests.RunAsync_ThreadPoolBacklogAfterExitDoesNotDiscardCompletedCapture`
runs on all OSes in a non-parallel collection. It runs a real process;
after exit, the capture queues behind a deterministic thread-pool
backlog that releases on cancellation or after 4s.
- Before the fix: **fails 5/5** (`Expected "captured", Actual ""`, about
2.1–2.6s).
  - After the fix: **passes 5/5**.
- **Interruption test**
`RunAsync_InterruptedDuringBackloggedDrainReportsInterruption`. While
the pool is backlogged, the caller cancels or the 3s deadline fires. It
asserts a `Cancelled` / `TimedOut` result with exit code -1. It covers
three cases:
  - **Caller cancels, capture returns buffered output on cancellation.**
- **Caller cancels, capture ignores cancellation** and stays queued
behind the backlog.
  - **Deadline fires, capture returns buffered output on cancellation.**
- The backlog starts from the runner's own thread as it enters the
post-exit drain. The hook is a new debug log line, `Process {Pid}
exited; draining its output.`, so the runner's exit continuation can
never be queued behind it.
- For caller cancellation, the backlog is held independently of the
token. A dedicated thread cancels as soon as the drain logs that its
bound expired and it is probing the backlogged pool (`checking the
thread pool for a backlog`), so the cancel always lands in the probe
wait, with no timers involved. The runner's new `Output drain for
process {Pid} was interrupted.` debug log records, inline and with no
pool worker, that the interruption was reported **while the pool was
still held**. The backlog is released only after that, because the
bounded cleanup afterwards is timer-driven.
- The deadline is the runner's own timer, which a starved pool can
delay, so that case releases the backlog once the token fires and checks
only the classification.
- It fails on main and on this PR's first revision. It also fails
deterministically (5/5) with a probe wait that ignores cancellation.
  - It passes 10/10 locally, plus 4/4 under full CPU load.
- **Real-pipe repro** on Linux: `GitCommandRunner.RunProcessAsync` on a
real `/bin/sh` `printf`, while the pool is flooded.
- Before the fix: 6/10 `RepositoryUnavailable` (drain TIMEOUT at about
2.0s, capture completes right after cancel).
  - After the fix: 0/10.
- **Related suites** after rebasing on main
(`ProcessCaptureRunnerTests`, `ConfigSourceServiceGitTests`,
`GitCommandRunnerTests`, `PeerInstallProbeTests`, `GitClientTests`,
`ConfigurationHelperTests`): 279 passed, 1 skipped (Windows-only). A 20×
stress of the regression test plus
`RunProcess_DescendantHoldingOutputCannotBlockShutdown` passed 20/20.
- **Real CLI:** a locally built `aspire update --config-source
--non-interactive --yes` against a real local git config source exited 0
with "configuration updated" and wrote `appliedVersion` and the docs
settings.

Fixes #21625

## Checklist

- Is this feature complete?
  - [x] Yes. Ready to ship.
  - [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
  - [x] Yes
  - [ ] No
- Did you add public API?
  - [ ] Yes
  - [x] No
- Does the change make any security assumptions or guarantees?
  - [ ] Yes
  - [x] No

**🤖 Posted by AspireAgent (AI Agent)**

---------

Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Mitch Denny <midenn@microsoft.com>
Resolve ProcessCaptureRunner conflict: keep main's thread-pool-aware
post-exit drain (#21626) and, when it genuinely gives up, cancel the reads
and keep the partial capture flagged OutputDrainTimedOut.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copy link
Copy Markdown
Collaborator Author

Merged main into this branch to resolve a conflict in ProcessCaptureRunner.RunAsync with #21626 (thread-pool-aware post-exit drain).

Resolution keeps both intents:

  • Main's DrainAfterExitAsync stays as is, so a backlogged pool no longer causes a false drain timeout, and deadline/cancellation during the drain still report TimedOut/Cancelled.
  • When the drain genuinely gives up (bound expires again on a responsive pool), it now returns a timed-out signal instead of an empty capture. RunAsync then cancels the reads, keeps the partial capture, and flags the result OutputDrainTimedOut as this PR intended. PeerInstallProbe accepts that; Git and ConfigurationHelper still reject it.

Locally, PeerInstallProbeTests, ProcessCaptureRunnerTests, ConfigSourceServiceGitTests, GitCommandRunnerTests, ConfigurationHelperTests and GitClientTests give 282 passed, 1 skipped (Windows-only). The new regression test plus RunProcess_DescendantHoldingOutputCannotBlockShutdown passed 5/5 repeated runs.

🤖 Posted by AspireAgent (AI Agent)

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Tests selector

2 / 100 PR test projects · 4 PR jobs, from 3 changed files.

Selected PR test projects (2 / 100)

Aspire.Cli.EndToEnd.Tests, Aspire.Cli.Tests

Selected PR jobs (4)

cli-starter-validation, extension-e2e, polyglot, typescript-api-compat


How these were chosen — grouped by what changed

📦 affected project Aspire.Cli
→ 1 test: Aspire.Cli.EndToEnd.Tests

🧪 tests/Aspire.Cli.Tests/Acquisition/PeerInstallProbeTests.cs (changed test)
→ 1 directly: Aspire.Cli.Tests

Job reasons

Job Triggered by
cli-starter-validation affected project Aspire.Cli
extension-e2e • src/Aspire.Cli/Acquisition/PeerInstallProbe.cs, src/Aspire.Cli/Utils/ProcessCaptureRunner.cs
• affected project Aspire.Cli
polyglot affected project Aspire.Cli
typescript-api-compat affected project Aspire.Cli

Selection computed for commit f47bb74.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the change to preserve captured CLI output after a post-exit drain timeout.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@radical

Ankit Jain (radical) commented Nov 8, 2026 •

Copy link
Copy Markdown
Member

[automated] Inconclusive: We did not observe the original failure in nine captured before-fix suites. These runs don’t establish whether the PR’s fix prevents it.

The original Windows failure reported:

Peer produced no usable output (and --version fallback).

Original-signature failures / captured full-suite runs:

Platform Before the fix With the PR’s fix
Windows x64 0/3 0/2; one unknown
Linux x64 0/3 0/3
macOS arm64 0/3 0/3

The workflow run testing the PR’s changes was marked failed because one Windows runner lost contact with GitHub. No steps, log, or test result were retained, so whether its tests ran is unknown. The other eight captured suites had no test failures.

Evidence caveat: The result checker rejected all 17 captured suites because a separate test included a random GUID in its name. A later offline audit supports the observations above, but none qualified for the original comparison.

Coverage gaps: The new regression passed in six Linux/macOS suites but skipped both captured Windows suites. It uses a 30-second timeout rather than the production probe’s default five seconds.

The historical test targeted .NET 10; these suites targeted .NET 11. The before-fix source already included a separate output-drain fix. The historical log does not establish the failure’s cause.

Run before the fix · Run with the PR’s fix

Test scope and revisions
  • Each captured runner executed the full regular Aspire.Cli.Tests project, excluding quarantined, outer-loop, and failing-category tests.
  • Tested sources: before the fix and with the PR’s fix. The workflow-run heads were separate from these tested source revisions.

Posted by AnkitAgent

@radical
Ankit Jain (radical) merged commit 1664086 into main Nov 9, 2026
430 of 434 checks passed
@microsoft-github-policy-service microsoft-github-policy-service Bot added this to the 17.1 milestone Nov 9, 2026
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

✅ No documentation update needed.

Skipped: proven false positive; no docs change needed.

  • Branch: docs_required with a false-positive signal.
  • Triggered signal (1): pr_body_has_cli_flag_mention. The only evidence is the PR body text "Peer produced no usable output (and --version fallback)", which quotes a CI test failure message. It does not describe a new Aspire CLI flag.
  • The PR is an internal reliability fix. ProcessCaptureRunner now keeps already-captured output when the post-exit drain bound expires. It adds an internal ProcessCaptureFailureKind.OutputDrainTimedOut and changes PeerInstallProbe handling and the test logger level.
  • No public commands, options, output formats, or documented behavior changed. This is an internal bug fix.

Mitch Denny (mitchdenny) pushed a commit that referenced this pull request Nov 10, 2026
…d expires (#21556)

## Description

`PeerInstallProbeTests.ProbeAsync_PeerEmitsArrayWithNonObjectFirstElement_FallsBackToVersion("[\"string\"]")`
failed once on Windows with `Peer produced no usable output (and
--version fallback)`. That run was [35003412018 / job
104497909578](https://github.com/microsoft/aspire/actions/runs/35003412018/job/104497909578).
The test took **4.3s**. Passing runs take a median of 0.154s and a max
of 0.51s (n=60, from TRX of 15 recent main Cli-windows runs, 300/300
PeerInstallProbe passes).

### Root cause

`ProcessCaptureRunner.RunAsync` handles a process that exits normally
like this:
- It waits up to a fixed 2s (`s_postExitCaptureWaitBound`) for the
stdout/stderr capture to reach EOF.
- If that bound expires, `SwallowCaptureAsync` returns
**`createEmptyCapture()`**. That throws away everything the reader had
already buffered. Only after that does it cancel the reads.

So a peer that printed `9.0.0` and exited 0 could come back as exit 0
with empty stdout. `ExtractVersionLine` then returns null and the probe
reports "no usable output". EOF arrives late when a descendant still
holds the inherited pipe handle, or when reader continuations are
delayed under CI load. The extra 2s matches the 4.3s failure duration.

The probe's Debug diagnostics, such as "Timed out waiting 2s for process
output capture", were missing from the CI log. The test's
`LoggerFactory` used the default Information minimum level, so they were
filtered out before reaching the xunit provider.

### Fix

When the post-exit drain bound expires, `RunAsync` now cancels the reads
and *then* collects the capture, waiting at most the 250ms post-kill
bound. It no longer abandons the capture. The result carries the real
exit code, the capture, and a new
`ProcessCaptureFailureKind.OutputDrainTimedOut`:
- `PeerInstallProbe` treats `OutputDrainTimedOut` as success.
`ReadCappedAsync` returns partial output on cancellation, so the version
the peer wrote before exiting is kept. The existing parsers still reject
unusable output.
- `GitCommandRunner` and `ConfigurationHelper` reject any non-null
`FailureKind`, so an incomplete drain still fails as before
(`RepositoryUnavailable` / settings write refused). This keeps
`ConfigSourceServiceGitTests.RunProcess_DescendantHoldingOutputCannotBlockShutdown`
passing. The first revision, without the flag, broke that test in CI,
which is why the flag exists.
- The happy path doesn't change. The final `timeoutCts.Cancel()` still
guarantees no capture task outlives the call.

I also changed the test logger factory to
`SetMinimumLevel(LogLevel.Trace)` so future failures include the probe's
Debug logs.

### Regression test and evidence

The new test is
`ProbeAsync_VersionFallbackDescendantHoldsStdoutOpen_KeepsCapturedVersion`
(POSIX only). The fake peer:
- fails `doctor`;
- on `--version`, prints `9.0.0`, leaves a background `sleep 60 &` that
inherits stdout, and exits 0.

This makes the late-EOF condition deterministic. The test runs real
processes through the real `ProcessCaptureRunner`; nothing is mocked.

| | Result |
|---|---|
| Before fix (Linux) | **Fails** after 2.1s. The log shows `Timed out
waiting 2s for process output capture` and the probe returns no version.
|
| After fix (Linux) | **10/10 pass**, about 2.08s each (the expected
drain bound, then cancellation). |
| `PeerInstallProbeTests`, `ConfigSourceService*`, `GitCommandRunner*`,
`ConfigurationHelper*` and `ProcessCaptureRunner*` classes, after fix |
340 total: 339 passed, 1 skipped (Windows-only), 0 failed. These run
real `git` and real descendant processes. |
| New test plus `RunProcess_DescendantHoldingOutputCannotBlockShutdown`,
repeated | 5 × 5 = 25/25 passed |

**Caveat:** the Windows failure left no probe diagnostics, so I can't
prove the late EOF from that one log. One alternative remains possible:
the fake peer's `echo %* | findstr` failing transiently, which exits 127
and produces the same message. The 2s-shaped extra duration points to
the drain path, and the fix closes that path either way.

Fixes #20147

## Checklist
- Is this feature complete?
  - [x] Yes. Ready to ship.
  - [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
  - [x] Yes
  - [ ] No
- Did you add public API?
  - [ ] Yes
  - [x] No
- Does the change make any security assumptions or guarantees?
  - [ ] Yes
  - [x] No
- Does the change require an update in our Aspire docs?
  - [ ] Yes
  - [x] No

**🤖 Posted by AspireAgent (AI Agent)**

---------

Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit 1664086)
Mitch Denny (mitchdenny) pushed a commit that referenced this pull request Nov 10, 2026
…d expires (#21556)

## Description

`PeerInstallProbeTests.ProbeAsync_PeerEmitsArrayWithNonObjectFirstElement_FallsBackToVersion("[\"string\"]")`
failed once on Windows with `Peer produced no usable output (and
--version fallback)`. That run was [35003412018 / job
104497909578](https://github.com/microsoft/aspire/actions/runs/35003412018/job/104497909578).
The test took **4.3s**. Passing runs take a median of 0.154s and a max
of 0.51s (n=60, from TRX of 15 recent main Cli-windows runs, 300/300
PeerInstallProbe passes).

### Root cause

`ProcessCaptureRunner.RunAsync` handles a process that exits normally
like this:
- It waits up to a fixed 2s (`s_postExitCaptureWaitBound`) for the
stdout/stderr capture to reach EOF.
- If that bound expires, `SwallowCaptureAsync` returns
**`createEmptyCapture()`**. That throws away everything the reader had
already buffered. Only after that does it cancel the reads.

So a peer that printed `9.0.0` and exited 0 could come back as exit 0
with empty stdout. `ExtractVersionLine` then returns null and the probe
reports "no usable output". EOF arrives late when a descendant still
holds the inherited pipe handle, or when reader continuations are
delayed under CI load. The extra 2s matches the 4.3s failure duration.

The probe's Debug diagnostics, such as "Timed out waiting 2s for process
output capture", were missing from the CI log. The test's
`LoggerFactory` used the default Information minimum level, so they were
filtered out before reaching the xunit provider.

### Fix

When the post-exit drain bound expires, `RunAsync` now cancels the reads
and *then* collects the capture, waiting at most the 250ms post-kill
bound. It no longer abandons the capture. The result carries the real
exit code, the capture, and a new
`ProcessCaptureFailureKind.OutputDrainTimedOut`:
- `PeerInstallProbe` treats `OutputDrainTimedOut` as success.
`ReadCappedAsync` returns partial output on cancellation, so the version
the peer wrote before exiting is kept. The existing parsers still reject
unusable output.
- `GitCommandRunner` and `ConfigurationHelper` reject any non-null
`FailureKind`, so an incomplete drain still fails as before
(`RepositoryUnavailable` / settings write refused). This keeps
`ConfigSourceServiceGitTests.RunProcess_DescendantHoldingOutputCannotBlockShutdown`
passing. The first revision, without the flag, broke that test in CI,
which is why the flag exists.
- The happy path doesn't change. The final `timeoutCts.Cancel()` still
guarantees no capture task outlives the call.

I also changed the test logger factory to
`SetMinimumLevel(LogLevel.Trace)` so future failures include the probe's
Debug logs.

### Regression test and evidence

The new test is
`ProbeAsync_VersionFallbackDescendantHoldsStdoutOpen_KeepsCapturedVersion`
(POSIX only). The fake peer:
- fails `doctor`;
- on `--version`, prints `9.0.0`, leaves a background `sleep 60 &` that
inherits stdout, and exits 0.

This makes the late-EOF condition deterministic. The test runs real
processes through the real `ProcessCaptureRunner`; nothing is mocked.

| | Result |
|---|---|
| Before fix (Linux) | **Fails** after 2.1s. The log shows `Timed out
waiting 2s for process output capture` and the probe returns no version.
|
| After fix (Linux) | **10/10 pass**, about 2.08s each (the expected
drain bound, then cancellation). |
| `PeerInstallProbeTests`, `ConfigSourceService*`, `GitCommandRunner*`,
`ConfigurationHelper*` and `ProcessCaptureRunner*` classes, after fix |
340 total: 339 passed, 1 skipped (Windows-only), 0 failed. These run
real `git` and real descendant processes. |
| New test plus `RunProcess_DescendantHoldingOutputCannotBlockShutdown`,
repeated | 5 × 5 = 25/25 passed |

**Caveat:** the Windows failure left no probe diagnostics, so I can't
prove the late EOF from that one log. One alternative remains possible:
the fake peer's `echo %* | findstr` failing transiently, which exits 127
and produces the same message. The 2s-shaped extra duration points to
the drain path, and the fix closes that path either way.

Fixes #20147

## Checklist
- Is this feature complete?
  - [x] Yes. Ready to ship.
  - [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
  - [x] Yes
  - [ ] No
- Did you add public API?
  - [ ] Yes
  - [x] No
- Does the change make any security assumptions or guarantees?
  - [ ] Yes
  - [x] No
- Does the change require an update in our Aspire docs?
  - [ ] Yes
  - [x] No

**🤖 Posted by AspireAgent (AI Agent)**

---------

Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit 1664086)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[CI Failure] Flaky: PeerInstallProbeTests fallback-to-version test fails on Windows with generated batch script

3 participants