You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Don't discard completed process output when the thread pool is backlogged after exit - #21626
ConfigSourceServiceGitTests.AcquireAndCheck_MatchRepositoryObjectFormat fails on Windows Cli jobs with The service did not persist the Git version check. Over 7 days of ci.yml attempt-1 jobs it failed in 2 of 307 Windows PR jobs, 0 of 56 Windows main jobs, and 0 Linux/macOS jobs. Details are in #21625.
Root cause
In both failing logs, the check-phase git ls-remote logs Git process completed after 00:00:03.58; exit code 0 and no output drained line. Other git calls in the same test take 0.05–0.27s.
ProcessCaptureRunner waits at most 2s (captureTask.WaitAsync(2s)) for the capture after the process exits, so descendants holding inherited pipes can't block callers. The bound's timer fires without a thread-pool thread, but the stdout/stderr read continuations need one. On Windows, ReadAsync on the synchronous redirected pipes goes through the pool even for already-buffered data.
When the test host's pool is backlogged, the 2s bound expires while the reads of an exited, fully written git are still queued. The capture is discarded and GitCommandRunner maps it to RepositoryUnavailable. The non-forced check swallows that, so check.json is never written.
Fix
When the post-exit bound expires, ProcessCaptureRunner now measures how long a newly queued global work item takes to run.
If the pool runs it promptly (under 250ms), the reads aren't starved; output really is being held open. The runner gives up exactly as before, so the descendant-holding-pipe contract (RunProcess_DescendantHoldingOutputCannotBlockShutdown) is unchanged.
If the pool is backlogged, the runner keeps waiting for the capture until the pool can schedule reads. Every post-exit wait, including the probe, races capture completion against the operation token, so none of them waits for the pool to recover. If the deadline or the caller's cancellation fires before the output has drained, the result is TimedOut or Cancelled like the pre-exit path, not a successful run with partial output.
There are no sleeps or retries of the process, and no assertions were weakened. The spec in docs/specs/cli-config-sources.md is updated to match.
This is complementary to #21556. That PR keeps partial output (flagged OutputDrainTimedOut) for PeerInstallProbe, but it intentionally still fails Git on a drain timeout. This PR avoids that false drain timeout when the output is actually complete.
Evidence
Regression testProcessCaptureRunnerTests.RunAsync_ThreadPoolBacklogAfterExitDoesNotDiscardCompletedCapture runs on all OSes in a non-parallel collection. It runs a real process; after exit, the capture queues behind a deterministic thread-pool backlog that releases on cancellation or after 4s.
Before the fix: fails 5/5 (Expected "captured", Actual "", about 2.1–2.6s).
After the fix: passes 5/5.
Interruption testRunAsync_InterruptedDuringBackloggedDrainReportsInterruption. While the pool is backlogged, the caller cancels or the 3s deadline fires. It asserts a Cancelled / TimedOut result with exit code -1. It covers three cases:
Caller cancels, capture returns buffered output on cancellation.
Caller cancels, capture ignores cancellation and stays queued behind the backlog.
Deadline fires, capture returns buffered output on cancellation.
The backlog starts from the runner's own thread as it enters the post-exit drain. The hook is a new debug log line, Process {Pid} exited; draining its output., so the runner's exit continuation can never be queued behind it.
For caller cancellation, the backlog is held independently of the token. A dedicated thread cancels as soon as the drain logs that its bound expired and it is probing the backlogged pool (checking the thread pool for a backlog), so the cancel always lands in the probe wait, with no timers involved. The runner's new Output drain for process {Pid} was interrupted. debug log records, inline and with no pool worker, that the interruption was reported while the pool was still held. The backlog is released only after that, because the bounded cleanup afterwards is timer-driven.
The deadline is the runner's own timer, which a starved pool can delay, so that case releases the backlog once the token fires and checks only the classification.
It fails on main and on this PR's first revision. It also fails deterministically (5/5) with a probe wait that ignores cancellation.
It passes 10/10 locally, plus 4/4 under full CPU load.
Real-pipe repro on Linux: GitCommandRunner.RunProcessAsync on a real /bin/shprintf, while the pool is flooded.
Before the fix: 6/10 RepositoryUnavailable (drain TIMEOUT at about 2.0s, capture completes right after cancel).
After the fix: 0/10.
Related suites after rebasing on main (ProcessCaptureRunnerTests, ConfigSourceServiceGitTests, GitCommandRunnerTests, PeerInstallProbeTests, GitClientTests, ConfigurationHelperTests): 279 passed, 1 skipped (Windows-only). A 20× stress of the regression test plus RunProcess_DescendantHoldingOutputCannotBlockShutdown passed 20/20.
Real CLI: a locally built aspire update --config-source --non-interactive --yes against a real local git config source exited 0 with "configuration updated" and wrote appliedVersion and the docs settings.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Critical issues leave pool-probe waits unbounded by cancellation or deadlines and allow cancellation or timeout during extended draining to be reported as success.
This PR addresses intermittent CLI Git-check failures caused by discarding completed process output when thread-pool backlog delays capture continuations.
Changes:
Measures thread-pool scheduling delay before abandoning post-exit output capture.
Adds a regression test that deliberately backlogs the thread pool.
Updates the configuration-source specification to describe backlog-aware draining.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
The pool-recovery branch can discard pending output capture and incorrectly report RepositoryUnavailable for a completed Git process; it needs a cancellation-aware bounded drain and regression coverage around the initial drain deadline.
No open issues 🎉
1 previously missed issue
In code that hasn't changed since last review
Recovery probe can discard output before capture finishes
If the pool recovers just before the probe is queued, the probe can finish promptly while GitCommandRunner.ReadOutputAsync is still draining buffered output or awaiting its next read; starting earlier does not guarantee that capture finishes first. This branch then discards the capture and cancels its reads, so a completed Git process can still produce RepositoryUnavailable without a descendant holding the pipe. Allow a fresh, cancellation-aware bounded drain after the pool becomes responsive, and add a regression covering recovery around the initial drain deadline.
Windows Cli (attempt 1, job 124721834587) failed with one unrelated test:
GitCommandRunnerTests.ConfigureUnattendedSsh_VerifiesOpenSshAndAppliesBatchModeAfterUrlRewrites(rewritten: False, setting: "GIT_SSH_COMMAND"): Unattended Git requires a verified OpenSSH executable…, after 5.2s.
The two preceding git calls finished in 0.04s and 0.13s. The remaining ~5s matches the 5s timeout of the ssh -V probe. The runner heartbeat at that moment showed CPU at 86.5%, with git at 99% CPU.
This PR can't turn a successful probe into this failure. The first 2s post-exit wait is unchanged. Previously an expired bound always returned a null capture, which fails this check. Now the runner either recovers the complete capture or fails at the same 5s deadline.
The test passes locally with this change (GitCommandRunnerTests is in the 279 passed, 1 skipped run).
This is a load-related probe timeout, so I'm rerunning the failed job once.
Update: CI attempt 1 on d493490 also failed macOS Cli, in this PR's own new assertion. RunAsync_InterruptedDuringBackloggedDrainReportsInterruption(cancelByCaller: True) returned only after the backlog was released.
Cause: in each timer-queue pass .NET fires only the first expired timer on the timer thread and queues the rest to the thread pool. The test's 4s CancelAfter coincided with the drain's second 2s bound, so the cancellation itself could wait behind the test's backlog.
The interruption test no longer depends on timer delivery. The capture stays open until the token fires, like a reader returning buffered output on cancellation, and the backlog is released on that token. The outcome is deterministic whichever timer fires first.
The release-time assertion is removed.
The first post-exit wait now observes the token too, so an interruption is reported in every drain phase.
Before/after: the theory fails on both main and dac5c31 (Expected TimedOut / Actual null, Expected True / Actual False) and passes 5/5 locally now. Related suites: 278 passed, 1 skipped, 1 unrelated failure seen once (4ms PeerInstallProbeTests start-up, 0 in 15 re-runs).
The new push restarts CI, so the Windows ConfigureUnattendedSsh… rerun described above is superseded. If it recurs I'll handle it as a separate flake.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Pool recovery just after the initial bound can cause valid capture output to be discarded, and the interruption test does not verify completion before pool recovery.
If the pool recovers just after the 2s bound, the probe can finish on one worker in under 250ms while another worker is still completing the previously starved capture. This branch then discards the output and cancels the reads even though the delay came from the backlog, not an inherited pipe. Give the capture a bounded drain opportunity after the pool becomes responsive, and cover recovery just after the initial bound.
Rerunning one unrelated failure in run 41266068726: VS Code extension E2E (Windows, resource-debug-tools-negative) (job).
The test cancels through the VS Code invocation token and reports a stopped resource without invoking a debugger hit Timed out after 10000ms waiting for E2E control revision while the extension was polling aspire resource stop/aspire ps. That path doesn't use ProcessCaptureRunner, which is used only for git, the peer-install probe and configuration helper captures, so this PR doesn't touch it. All the Cli test jobs passed on ubuntu, windows and macOS. Rerunning the failed job once.
Update on run 41266068726: VS Code extension E2E (Windows, debug-startup-timeout) hung in Run extension E2E tests. It ran about 30 minutes with no progress; the shard usually takes about 6 minutes.
The same shard hung the same way on main in run 41234363151 (job 124653890239, ~47 min, no result), so this isn't caused by this PR. Every other check passed, including all the Cli test jobs.
I cancelled the run and am rerunning only its failed or cancelled jobs once: this shard and the resource-debug-tools-negative shard from my previous comment.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed to evaluate a cancellation race that can report partial output as successful and a timing-sensitive test that may fail under Windows CI load.
On 4855109, the new assertion failed once on ubuntu Cli (job): (cancelByCaller: True, captureObservesCancellation: True) completed only once the backlog was released, after 20s.
The cause was in the test, not the drain. The capture started the backlog after its own WaitForExitAsync plus Task.Yield. On Unix, the runner's own WaitForExitAsync continuation is queued separately to the pool, so it could still be waiting behind the backlog. The runner then sat in the pre-exit wait, which wasn't the path under test.
The runner logs Process {Pid} exited; draining its output. at debug level when it enters the drain. The test starts the backlog from that log callback, on the runner's own thread.
The capture that ignores cancellation explicitly queues behind the backlog on the global queue.
Results: 15/15 locally, 6/6 under full CPU load, and the full Cli suite. A probe wait that ignores cancellation still fails both caller cases at 20s.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because timing-dependent synchronization may let the regression test pass without exercising the intended post-exit drain behavior.
…rruption test deterministic
Timer callbacks after the first in a timer-queue pass are queued to the thread
pool, so a timer-driven cancellation can wait behind the test's backlog. The
interruption test now holds the capture open until the token fires and releases
the backlog on that token, so its outcome no longer depends on timer delivery.
The first post-exit wait also observes the token, so an interruption is reported
in every phase of the drain.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Hold the backlog independently of the operation token for caller
cancellation, cancel from a dedicated thread, and record completion inline
so the test fails if any post-exit wait ignores cancellation.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Starting the backlog from the capture could leave the runner's own exit
continuation queued behind it, so the runner sat in the pre-exit wait until
the pool recovered. Log drain entry and start the backlog from that hook.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Log when the post-exit drain is interrupted and when it starts probing the
thread pool. The caller-cancel variants now cancel from a dedicated thread
once the drain is probing the backlogged pool, and assert that the runner
reports the interruption while the backlog is still held, instead of
depending on timers that a starved pool can delay.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rebased onto current main to clear the conflict. The only conflict was the Git-acquisition paragraph in docs/specs/cli-config-sources.md: I kept main's rewritten text (symlink/size-limit changes from #21682) and re-added this PR's two sentences about the post-exit drain extension and its timeout/cancellation bound. The code and test commits applied unchanged. Locally, the ProcessCaptureRunner, ConfigSourceServiceGit, GitCommandRunner, PeerInstallProbe, GitClient and ConfigurationHelper tests pass (279 passed, 1 Windows-only skip).
Update: after the rebase, macOS CI failed RunAsync_InterruptedDuringBackloggedDrainReportsInterruption(cancelByCaller: True, captureObservesCancellation: True) because the interruption was only reported once the backlog released at 20s. The backlog can also starve the 2s drain bound's own timer, so the probe that triggers the caller's cancellation never ran. bb6f406 arms a dedicated-thread fallback at drain entry, so the cancel then lands in the bound's wait. The assertion is unchanged, and that path still has to report the interruption while the pool is held (verified locally by forcing it). This is on top of Mitch Denny (@mitchdenny)'s 0baa442.
Give output reads one final bounded, cancellation-aware drain after a responsive pool probe. Cover multi-step capture completion and cancellation during recovery.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because renewed thread-pool backlog during recovery draining can cause completed output to be discarded and repository unavailability to be reported before the operation deadline.
…d's timer
On macOS CI the backlog also starved the 2s drain bound's timer, so the probe
log that triggers the caller's cancellation only ran once the backlog was
released. Arm a dedicated-thread fallback at drain entry; if the probe is not
reached, the cancel lands in the bound's wait, which must observe it without
the pool just the same.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
Reviewed the recovery-drain fix and subsequent test cancellation fallback. No remaining code findings. CI currently has failures in Windows ARM64 starter validation and TypeScript/Blazor gateway E2E; those remain merge-validation items.
Proven false positive → no docs needed (internal bug fix).
Triggered signal: pr_body_has_cli_flag_mention. Evidence: the PR body mentions aspire update --config-source --non-interactive --yes only as a manual verification run, not as a new or changed flag.
The change is internal to ProcessCaptureRunner: it stops discarding completed process output when the thread pool is backlogged after the process exits. It restores the intended behavior (a transient Windows CI flake in a Git config-source check). No new options, strings, or public surface.
The implementation spec docs/specs/cli-config-sources.md was updated in the source repo and is not part of aspire.dev.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
area-clici-healthFixes CI flakiness or CI infrastructure (AspireAgent queue)
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
ConfigSourceServiceGitTests.AcquireAndCheck_MatchRepositoryObjectFormatfails on Windows Cli jobs withThe service did not persist the Git version check.Over 7 days of ci.yml attempt-1 jobs it failed in 2 of 307 Windows PR jobs, 0 of 56 Windows main jobs, and 0 Linux/macOS jobs. Details are in #21625.Root cause
In both failing logs, the check-phase
git ls-remotelogsGit process completed after 00:00:03.58; exit code 0and nooutput drainedline. Other git calls in the same test take 0.05–0.27s.ProcessCaptureRunnerwaits at most 2s (captureTask.WaitAsync(2s)) for the capture after the process exits, so descendants holding inherited pipes can't block callers. The bound's timer fires without a thread-pool thread, but the stdout/stderr read continuations need one. On Windows,ReadAsyncon the synchronous redirected pipes goes through the pool even for already-buffered data.When the test host's pool is backlogged, the 2s bound expires while the reads of an exited, fully written
gitare still queued. The capture is discarded andGitCommandRunnermaps it toRepositoryUnavailable. The non-forced check swallows that, socheck.jsonis never written.Fix
When the post-exit bound expires,
ProcessCaptureRunnernow measures how long a newly queued global work item takes to run.RunProcess_DescendantHoldingOutputCannotBlockShutdown) is unchanged.TimedOutorCancelledlike the pre-exit path, not a successful run with partial output.There are no sleeps or retries of the process, and no assertions were weakened. The spec in
docs/specs/cli-config-sources.mdis updated to match.This is complementary to #21556. That PR keeps partial output (flagged
OutputDrainTimedOut) forPeerInstallProbe, but it intentionally still fails Git on a drain timeout. This PR avoids that false drain timeout when the output is actually complete.Evidence
ProcessCaptureRunnerTests.RunAsync_ThreadPoolBacklogAfterExitDoesNotDiscardCompletedCaptureruns on all OSes in a non-parallel collection. It runs a real process; after exit, the capture queues behind a deterministic thread-pool backlog that releases on cancellation or after 4s.Expected "captured", Actual "", about 2.1–2.6s).RunAsync_InterruptedDuringBackloggedDrainReportsInterruption. While the pool is backlogged, the caller cancels or the 3s deadline fires. It asserts aCancelled/TimedOutresult with exit code -1. It covers three cases:Process {Pid} exited; draining its output., so the runner's exit continuation can never be queued behind it.checking the thread pool for a backlog), so the cancel always lands in the probe wait, with no timers involved. The runner's newOutput drain for process {Pid} was interrupted.debug log records, inline and with no pool worker, that the interruption was reported while the pool was still held. The backlog is released only after that, because the bounded cleanup afterwards is timer-driven.GitCommandRunner.RunProcessAsyncon a real/bin/shprintf, while the pool is flooded.RepositoryUnavailable(drain TIMEOUT at about 2.0s, capture completes right after cancel).ProcessCaptureRunnerTests,ConfigSourceServiceGitTests,GitCommandRunnerTests,PeerInstallProbeTests,GitClientTests,ConfigurationHelperTests): 279 passed, 1 skipped (Windows-only). A 20× stress of the regression test plusRunProcess_DescendantHoldingOutputCannotBlockShutdownpassed 20/20.aspire update --config-source --non-interactive --yesagainst a real local git config source exited 0 with "configuration updated" and wroteappliedVersionand the docs settings.Fixes #21625
Checklist
🤖 Posted by AspireAgent (AI Agent)