Repository navigation
Keep captured output when ProcessCaptureRunner's post-exit drain bound expires - #21556
Conversation
ProcessCaptureRunner waited a fixed 2s for the output capture after the process exited normally and, on timeout, discarded everything captured so far and returned an empty capture. A peer that printed its version and exited 0 could then be reported as "Peer produced no usable output" when stdout EOF arrived late (a descendant holding the inherited pipe, or delayed reader continuations under CI load). When the drain bound expires, cancel the reads instead and collect the capture, so captures that return partial output on cancellation (the PeerInstallProbe readers) keep what was already written. Callers whose captures throw on cancellation still get an empty capture as before. Also let the probe tests' logger emit Debug diagnostics, which the factory's default minimum level was filtering out. Fixes #20147 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21556Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21556" |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
🟡 Changes recommended
Post-cancellation output collection changes Git drain-timeout failures into success for exit-zero processes, breaking both parent-exit regression test cases.
1 open issue
What changed in this PR
Preserves buffered process output when the post-exit drain limit expires, addressing a possible cause of the peer-version probe failure tracked in #20147.
Changes:
- Cancels pending reads before collecting partial output within a short additional bound.
- Adds a POSIX regression test where a descendant keeps stdout open.
- Enables Trace-level test logging so probe diagnostics appear in failure output.
| File | Description |
|---|---|
| tests/Aspire.Cli.Tests/Acquisition/PeerInstallProbeTests.cs | Adds inherited-pipe regression coverage and enables detailed logging. |
| src/Aspire.Cli/Utils/ProcessCaptureRunner.cs | Collects capture results after cancelling reads when the drain limit expires. |
Review effort: Balanced
Callers such as GitCommandRunner must not treat output cut short by the post-exit drain bound as complete. Report it as ProcessCaptureFailureKind.OutputDrainTimedOut with the real exit code and the partial capture; PeerInstallProbe accepts it because its parsers already reject unusable output, while Git and ConfigurationHelper keep failing as before. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
|
CI note for head f987c01 (run 41186503338): I reran the failed jobs once. Both failures are in the known VS Code extension Windows E2E family (#21180/#20063, Deno Windows fixture):
Neither is related to this change. Git and ConfigurationHelper still treat an incomplete post-exit drain as a failure; only the peer probe accepts it. The Cli unit tests on ubuntu, macOS and Windows and the CLI archive E2E matrix all passed on this head. 🤖 Posted by AspireAgent (AI Agent) |
…gged after exit (#21626) ## Description `ConfigSourceServiceGitTests.AcquireAndCheck_MatchRepositoryObjectFormat` fails on Windows Cli jobs with `The service did not persist the Git version check.` Over 7 days of ci.yml attempt-1 jobs it failed in 2 of 307 Windows PR jobs, 0 of 56 Windows main jobs, and 0 Linux/macOS jobs. Details are in #21625. ### Root cause In both failing logs, the check-phase `git ls-remote` logs `Git process completed after 00:00:03.58; exit code 0` and no `output drained` line. Other git calls in the same test take 0.05–0.27s. `ProcessCaptureRunner` waits at most 2s (`captureTask.WaitAsync(2s)`) for the capture after the process exits, so descendants holding inherited pipes can't block callers. The bound's timer fires without a thread-pool thread, but the stdout/stderr read continuations need one. On Windows, `ReadAsync` on the synchronous redirected pipes goes through the pool even for already-buffered data. When the test host's pool is backlogged, the 2s bound expires while the reads of an exited, fully written `git` are still queued. The capture is discarded and `GitCommandRunner` maps it to `RepositoryUnavailable`. The non-forced check swallows that, so `check.json` is never written. ### Fix When the post-exit bound expires, `ProcessCaptureRunner` now measures how long a newly queued global work item takes to run. - If the pool runs it promptly (under 250ms), the reads aren't starved; output really is being held open. The runner gives up exactly as before, so the descendant-holding-pipe contract (`RunProcess_DescendantHoldingOutputCannotBlockShutdown`) is unchanged. - If the pool is backlogged, the runner keeps waiting for the capture until the pool can schedule reads. Every post-exit wait, including the probe, races capture completion against the operation token, so none of them waits for the pool to recover. If the deadline or the caller's cancellation fires before the output has drained, the result is `TimedOut` or `Cancelled` like the pre-exit path, not a successful run with partial output. There are no sleeps or retries of the process, and no assertions were weakened. The spec in `docs/specs/cli-config-sources.md` is updated to match. This is complementary to #21556. That PR keeps partial output (flagged `OutputDrainTimedOut`) for `PeerInstallProbe`, but it intentionally still fails Git on a drain timeout. This PR avoids that false drain timeout when the output is actually complete. ### Evidence - **Regression test** `ProcessCaptureRunnerTests.RunAsync_ThreadPoolBacklogAfterExitDoesNotDiscardCompletedCapture` runs on all OSes in a non-parallel collection. It runs a real process; after exit, the capture queues behind a deterministic thread-pool backlog that releases on cancellation or after 4s. - Before the fix: **fails 5/5** (`Expected "captured", Actual ""`, about 2.1–2.6s). - After the fix: **passes 5/5**. - **Interruption test** `RunAsync_InterruptedDuringBackloggedDrainReportsInterruption`. While the pool is backlogged, the caller cancels or the 3s deadline fires. It asserts a `Cancelled` / `TimedOut` result with exit code -1. It covers three cases: - **Caller cancels, capture returns buffered output on cancellation.** - **Caller cancels, capture ignores cancellation** and stays queued behind the backlog. - **Deadline fires, capture returns buffered output on cancellation.** - The backlog starts from the runner's own thread as it enters the post-exit drain. The hook is a new debug log line, `Process {Pid} exited; draining its output.`, so the runner's exit continuation can never be queued behind it. - For caller cancellation, the backlog is held independently of the token. A dedicated thread cancels as soon as the drain logs that its bound expired and it is probing the backlogged pool (`checking the thread pool for a backlog`), so the cancel always lands in the probe wait, with no timers involved. The runner's new `Output drain for process {Pid} was interrupted.` debug log records, inline and with no pool worker, that the interruption was reported **while the pool was still held**. The backlog is released only after that, because the bounded cleanup afterwards is timer-driven. - The deadline is the runner's own timer, which a starved pool can delay, so that case releases the backlog once the token fires and checks only the classification. - It fails on main and on this PR's first revision. It also fails deterministically (5/5) with a probe wait that ignores cancellation. - It passes 10/10 locally, plus 4/4 under full CPU load. - **Real-pipe repro** on Linux: `GitCommandRunner.RunProcessAsync` on a real `/bin/sh` `printf`, while the pool is flooded. - Before the fix: 6/10 `RepositoryUnavailable` (drain TIMEOUT at about 2.0s, capture completes right after cancel). - After the fix: 0/10. - **Related suites** after rebasing on main (`ProcessCaptureRunnerTests`, `ConfigSourceServiceGitTests`, `GitCommandRunnerTests`, `PeerInstallProbeTests`, `GitClientTests`, `ConfigurationHelperTests`): 279 passed, 1 skipped (Windows-only). A 20× stress of the regression test plus `RunProcess_DescendantHoldingOutputCannotBlockShutdown` passed 20/20. - **Real CLI:** a locally built `aspire update --config-source --non-interactive --yes` against a real local git config source exited 0 with "configuration updated" and wrote `appliedVersion` and the docs settings. Fixes #21625 ## Checklist - Is this feature complete? - [x] Yes. Ready to ship. - [ ] No. Follow-up changes expected. - Are you including unit tests for the changes and scenario tests if relevant? - [x] Yes - [ ] No - Did you add public API? - [ ] Yes - [x] No - Does the change make any security assumptions or guarantees? - [ ] Yes - [x] No **🤖 Posted by AspireAgent (AI Agent)** --------- Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Mitch Denny <midenn@microsoft.com>
Resolve ProcessCaptureRunner conflict: keep main's thread-pool-aware post-exit drain (#21626) and, when it genuinely gives up, cancel the reads and keep the partial capture flagged OutputDrainTimedOut. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Merged Resolution keeps both intents:
Locally, 🤖 Posted by AspireAgent (AI Agent) |
Tests selector2 / 100 PR test projects · 4 PR jobs, from 3 changed files. Selected PR test projects (2 / 100)
Selected PR jobs (4)
How these were chosen — grouped by what changed📦 affected project 🧪 Job reasons
Selection computed for commit |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
[automated] Inconclusive: We did not observe the original failure in nine captured before-fix suites. These runs don’t establish whether the PR’s fix prevents it. The original Windows failure reported: Original-signature failures / captured full-suite runs:
The workflow run testing the PR’s changes was marked failed because one Windows runner lost contact with GitHub. No steps, log, or test result were retained, so whether its tests ran is unknown. The other eight captured suites had no test failures. Evidence caveat: The result checker rejected all 17 captured suites because a separate test included a random GUID in its name. A later offline audit supports the observations above, but none qualified for the original comparison. Coverage gaps: The new regression passed in six Linux/macOS suites but skipped both captured Windows suites. It uses a 30-second timeout rather than the production probe’s default five seconds. The historical test targeted .NET 10; these suites targeted .NET 11. The before-fix source already included a separate output-drain fix. The historical log does not establish the failure’s cause. Run before the fix · Run with the PR’s fix Test scope and revisions
Posted by AnkitAgent |
|
✅ No documentation update needed. Skipped: proven false positive; no docs change needed.
|
…d expires (#21556) ## Description `PeerInstallProbeTests.ProbeAsync_PeerEmitsArrayWithNonObjectFirstElement_FallsBackToVersion("[\"string\"]")` failed once on Windows with `Peer produced no usable output (and --version fallback)`. That run was [35003412018 / job 104497909578](https://github.com/microsoft/aspire/actions/runs/35003412018/job/104497909578). The test took **4.3s**. Passing runs take a median of 0.154s and a max of 0.51s (n=60, from TRX of 15 recent main Cli-windows runs, 300/300 PeerInstallProbe passes). ### Root cause `ProcessCaptureRunner.RunAsync` handles a process that exits normally like this: - It waits up to a fixed 2s (`s_postExitCaptureWaitBound`) for the stdout/stderr capture to reach EOF. - If that bound expires, `SwallowCaptureAsync` returns **`createEmptyCapture()`**. That throws away everything the reader had already buffered. Only after that does it cancel the reads. So a peer that printed `9.0.0` and exited 0 could come back as exit 0 with empty stdout. `ExtractVersionLine` then returns null and the probe reports "no usable output". EOF arrives late when a descendant still holds the inherited pipe handle, or when reader continuations are delayed under CI load. The extra 2s matches the 4.3s failure duration. The probe's Debug diagnostics, such as "Timed out waiting 2s for process output capture", were missing from the CI log. The test's `LoggerFactory` used the default Information minimum level, so they were filtered out before reaching the xunit provider. ### Fix When the post-exit drain bound expires, `RunAsync` now cancels the reads and *then* collects the capture, waiting at most the 250ms post-kill bound. It no longer abandons the capture. The result carries the real exit code, the capture, and a new `ProcessCaptureFailureKind.OutputDrainTimedOut`: - `PeerInstallProbe` treats `OutputDrainTimedOut` as success. `ReadCappedAsync` returns partial output on cancellation, so the version the peer wrote before exiting is kept. The existing parsers still reject unusable output. - `GitCommandRunner` and `ConfigurationHelper` reject any non-null `FailureKind`, so an incomplete drain still fails as before (`RepositoryUnavailable` / settings write refused). This keeps `ConfigSourceServiceGitTests.RunProcess_DescendantHoldingOutputCannotBlockShutdown` passing. The first revision, without the flag, broke that test in CI, which is why the flag exists. - The happy path doesn't change. The final `timeoutCts.Cancel()` still guarantees no capture task outlives the call. I also changed the test logger factory to `SetMinimumLevel(LogLevel.Trace)` so future failures include the probe's Debug logs. ### Regression test and evidence The new test is `ProbeAsync_VersionFallbackDescendantHoldsStdoutOpen_KeepsCapturedVersion` (POSIX only). The fake peer: - fails `doctor`; - on `--version`, prints `9.0.0`, leaves a background `sleep 60 &` that inherits stdout, and exits 0. This makes the late-EOF condition deterministic. The test runs real processes through the real `ProcessCaptureRunner`; nothing is mocked. | | Result | |---|---| | Before fix (Linux) | **Fails** after 2.1s. The log shows `Timed out waiting 2s for process output capture` and the probe returns no version. | | After fix (Linux) | **10/10 pass**, about 2.08s each (the expected drain bound, then cancellation). | | `PeerInstallProbeTests`, `ConfigSourceService*`, `GitCommandRunner*`, `ConfigurationHelper*` and `ProcessCaptureRunner*` classes, after fix | 340 total: 339 passed, 1 skipped (Windows-only), 0 failed. These run real `git` and real descendant processes. | | New test plus `RunProcess_DescendantHoldingOutputCannotBlockShutdown`, repeated | 5 × 5 = 25/25 passed | **Caveat:** the Windows failure left no probe diagnostics, so I can't prove the late EOF from that one log. One alternative remains possible: the fake peer's `echo %* | findstr` failing transiently, which exits 127 and produces the same message. The 2s-shaped extra duration points to the drain path, and the fix closes that path either way. Fixes #20147 ## Checklist - Is this feature complete? - [x] Yes. Ready to ship. - [ ] No. Follow-up changes expected. - Are you including unit tests for the changes and scenario tests if relevant? - [x] Yes - [ ] No - Did you add public API? - [ ] Yes - [x] No - Does the change make any security assumptions or guarantees? - [ ] Yes - [x] No - Does the change require an update in our Aspire docs? - [ ] Yes - [x] No **🤖 Posted by AspireAgent (AI Agent)** --------- Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 1664086)
…d expires (#21556) ## Description `PeerInstallProbeTests.ProbeAsync_PeerEmitsArrayWithNonObjectFirstElement_FallsBackToVersion("[\"string\"]")` failed once on Windows with `Peer produced no usable output (and --version fallback)`. That run was [35003412018 / job 104497909578](https://github.com/microsoft/aspire/actions/runs/35003412018/job/104497909578). The test took **4.3s**. Passing runs take a median of 0.154s and a max of 0.51s (n=60, from TRX of 15 recent main Cli-windows runs, 300/300 PeerInstallProbe passes). ### Root cause `ProcessCaptureRunner.RunAsync` handles a process that exits normally like this: - It waits up to a fixed 2s (`s_postExitCaptureWaitBound`) for the stdout/stderr capture to reach EOF. - If that bound expires, `SwallowCaptureAsync` returns **`createEmptyCapture()`**. That throws away everything the reader had already buffered. Only after that does it cancel the reads. So a peer that printed `9.0.0` and exited 0 could come back as exit 0 with empty stdout. `ExtractVersionLine` then returns null and the probe reports "no usable output". EOF arrives late when a descendant still holds the inherited pipe handle, or when reader continuations are delayed under CI load. The extra 2s matches the 4.3s failure duration. The probe's Debug diagnostics, such as "Timed out waiting 2s for process output capture", were missing from the CI log. The test's `LoggerFactory` used the default Information minimum level, so they were filtered out before reaching the xunit provider. ### Fix When the post-exit drain bound expires, `RunAsync` now cancels the reads and *then* collects the capture, waiting at most the 250ms post-kill bound. It no longer abandons the capture. The result carries the real exit code, the capture, and a new `ProcessCaptureFailureKind.OutputDrainTimedOut`: - `PeerInstallProbe` treats `OutputDrainTimedOut` as success. `ReadCappedAsync` returns partial output on cancellation, so the version the peer wrote before exiting is kept. The existing parsers still reject unusable output. - `GitCommandRunner` and `ConfigurationHelper` reject any non-null `FailureKind`, so an incomplete drain still fails as before (`RepositoryUnavailable` / settings write refused). This keeps `ConfigSourceServiceGitTests.RunProcess_DescendantHoldingOutputCannotBlockShutdown` passing. The first revision, without the flag, broke that test in CI, which is why the flag exists. - The happy path doesn't change. The final `timeoutCts.Cancel()` still guarantees no capture task outlives the call. I also changed the test logger factory to `SetMinimumLevel(LogLevel.Trace)` so future failures include the probe's Debug logs. ### Regression test and evidence The new test is `ProbeAsync_VersionFallbackDescendantHoldsStdoutOpen_KeepsCapturedVersion` (POSIX only). The fake peer: - fails `doctor`; - on `--version`, prints `9.0.0`, leaves a background `sleep 60 &` that inherits stdout, and exits 0. This makes the late-EOF condition deterministic. The test runs real processes through the real `ProcessCaptureRunner`; nothing is mocked. | | Result | |---|---| | Before fix (Linux) | **Fails** after 2.1s. The log shows `Timed out waiting 2s for process output capture` and the probe returns no version. | | After fix (Linux) | **10/10 pass**, about 2.08s each (the expected drain bound, then cancellation). | | `PeerInstallProbeTests`, `ConfigSourceService*`, `GitCommandRunner*`, `ConfigurationHelper*` and `ProcessCaptureRunner*` classes, after fix | 340 total: 339 passed, 1 skipped (Windows-only), 0 failed. These run real `git` and real descendant processes. | | New test plus `RunProcess_DescendantHoldingOutputCannotBlockShutdown`, repeated | 5 × 5 = 25/25 passed | **Caveat:** the Windows failure left no probe diagnostics, so I can't prove the late EOF from that one log. One alternative remains possible: the fake peer's `echo %* | findstr` failing transiently, which exits 127 and produces the same message. The 2s-shaped extra duration points to the drain path, and the fix closes that path either way. Fixes #20147 ## Checklist - Is this feature complete? - [x] Yes. Ready to ship. - [ ] No. Follow-up changes expected. - Are you including unit tests for the changes and scenario tests if relevant? - [x] Yes - [ ] No - Did you add public API? - [ ] Yes - [x] No - Does the change make any security assumptions or guarantees? - [ ] Yes - [x] No - Does the change require an update in our Aspire docs? - [ ] Yes - [x] No **🤖 Posted by AspireAgent (AI Agent)** --------- Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 1664086)

Description
PeerInstallProbeTests.ProbeAsync_PeerEmitsArrayWithNonObjectFirstElement_FallsBackToVersion("[\"string\"]")failed once on Windows withPeer produced no usable output (and --version fallback). That run was 35003412018 / job 104497909578. The test took 4.3s. Passing runs take a median of 0.154s and a max of 0.51s (n=60, from TRX of 15 recent main Cli-windows runs, 300/300 PeerInstallProbe passes).Root cause
ProcessCaptureRunner.RunAsynchandles a process that exits normally like this:s_postExitCaptureWaitBound) for the stdout/stderr capture to reach EOF.SwallowCaptureAsyncreturnscreateEmptyCapture(). That throws away everything the reader had already buffered. Only after that does it cancel the reads.So a peer that printed
9.0.0and exited 0 could come back as exit 0 with empty stdout.ExtractVersionLinethen returns null and the probe reports "no usable output". EOF arrives late when a descendant still holds the inherited pipe handle, or when reader continuations are delayed under CI load. The extra 2s matches the 4.3s failure duration.The probe's Debug diagnostics, such as "Timed out waiting 2s for process output capture", were missing from the CI log. The test's
LoggerFactoryused the default Information minimum level, so they were filtered out before reaching the xunit provider.Fix
When the post-exit drain bound expires,
RunAsyncnow cancels the reads and then collects the capture, waiting at most the 250ms post-kill bound. It no longer abandons the capture. The result carries the real exit code, the capture, and a newProcessCaptureFailureKind.OutputDrainTimedOut:PeerInstallProbetreatsOutputDrainTimedOutas success.ReadCappedAsyncreturns partial output on cancellation, so the version the peer wrote before exiting is kept. The existing parsers still reject unusable output.GitCommandRunnerandConfigurationHelperreject any non-nullFailureKind, so an incomplete drain still fails as before (RepositoryUnavailable/ settings write refused). This keepsConfigSourceServiceGitTests.RunProcess_DescendantHoldingOutputCannotBlockShutdownpassing. The first revision, without the flag, broke that test in CI, which is why the flag exists.timeoutCts.Cancel()still guarantees no capture task outlives the call.I also changed the test logger factory to
SetMinimumLevel(LogLevel.Trace)so future failures include the probe's Debug logs.Regression test and evidence
The new test is
ProbeAsync_VersionFallbackDescendantHoldsStdoutOpen_KeepsCapturedVersion(POSIX only). The fake peer:doctor;--version, prints9.0.0, leaves a backgroundsleep 60 &that inherits stdout, and exits 0.This makes the late-EOF condition deterministic. The test runs real processes through the real
ProcessCaptureRunner; nothing is mocked.Timed out waiting 2s for process output captureand the probe returns no version.PeerInstallProbeTests,ConfigSourceService*,GitCommandRunner*,ConfigurationHelper*andProcessCaptureRunner*classes, after fixgitand real descendant processes.RunProcess_DescendantHoldingOutputCannotBlockShutdown, repeatedCaveat: the Windows failure left no probe diagnostics, so I can't prove the late EOF from that one log. One alternative remains possible: the fake peer's
echo %* | findstrfailing transiently, which exits 127 and produces the same message. The 2s-shaped extra duration points to the drain path, and the fix closes that path either way.Fixes #20147
Checklist
🤖 Posted by AspireAgent (AI Agent)