Skip to content

Don't discard completed process output when the thread pool is backlogged after exit - #21626

Merged
David Fowler (davidfowl) merged 9 commits into
mainfrom
davidfowl/configsource-git-drain-starvation
Nov 3, 2026
Merged

David Fowler (davidfowl) merged 9 commits into
mainfrom
davidfowl/configsource-git-drain-starvation

Conversation

@davidfowl

@davidfowl David Fowler (davidfowl) commented Nov 1, 2026 •

Copy link
Copy Markdown
Collaborator

Description

ConfigSourceServiceGitTests.AcquireAndCheck_MatchRepositoryObjectFormat fails on Windows Cli jobs with The service did not persist the Git version check. Over 7 days of ci.yml attempt-1 jobs it failed in 2 of 307 Windows PR jobs, 0 of 56 Windows main jobs, and 0 Linux/macOS jobs. Details are in #21625.

Root cause

In both failing logs, the check-phase git ls-remote logs Git process completed after 00:00:03.58; exit code 0 and no output drained line. Other git calls in the same test take 0.05–0.27s.

ProcessCaptureRunner waits at most 2s (captureTask.WaitAsync(2s)) for the capture after the process exits, so descendants holding inherited pipes can't block callers. The bound's timer fires without a thread-pool thread, but the stdout/stderr read continuations need one. On Windows, ReadAsync on the synchronous redirected pipes goes through the pool even for already-buffered data.

When the test host's pool is backlogged, the 2s bound expires while the reads of an exited, fully written git are still queued. The capture is discarded and GitCommandRunner maps it to RepositoryUnavailable. The non-forced check swallows that, so check.json is never written.

Fix

When the post-exit bound expires, ProcessCaptureRunner now measures how long a newly queued global work item takes to run.

  • If the pool runs it promptly (under 250ms), the reads aren't starved; output really is being held open. The runner gives up exactly as before, so the descendant-holding-pipe contract (RunProcess_DescendantHoldingOutputCannotBlockShutdown) is unchanged.
  • If the pool is backlogged, the runner keeps waiting for the capture until the pool can schedule reads. Every post-exit wait, including the probe, races capture completion against the operation token, so none of them waits for the pool to recover. If the deadline or the caller's cancellation fires before the output has drained, the result is TimedOut or Cancelled like the pre-exit path, not a successful run with partial output.

There are no sleeps or retries of the process, and no assertions were weakened. The spec in docs/specs/cli-config-sources.md is updated to match.

This is complementary to #21556. That PR keeps partial output (flagged OutputDrainTimedOut) for PeerInstallProbe, but it intentionally still fails Git on a drain timeout. This PR avoids that false drain timeout when the output is actually complete.

Evidence

  • Regression test ProcessCaptureRunnerTests.RunAsync_ThreadPoolBacklogAfterExitDoesNotDiscardCompletedCapture runs on all OSes in a non-parallel collection. It runs a real process; after exit, the capture queues behind a deterministic thread-pool backlog that releases on cancellation or after 4s.
    • Before the fix: fails 5/5 (Expected "captured", Actual "", about 2.1–2.6s).
    • After the fix: passes 5/5.
  • Interruption test RunAsync_InterruptedDuringBackloggedDrainReportsInterruption. While the pool is backlogged, the caller cancels or the 3s deadline fires. It asserts a Cancelled / TimedOut result with exit code -1. It covers three cases:
    • Caller cancels, capture returns buffered output on cancellation.
    • Caller cancels, capture ignores cancellation and stays queued behind the backlog.
    • Deadline fires, capture returns buffered output on cancellation.
    • The backlog starts from the runner's own thread as it enters the post-exit drain. The hook is a new debug log line, Process {Pid} exited; draining its output., so the runner's exit continuation can never be queued behind it.
    • For caller cancellation, the backlog is held independently of the token. A dedicated thread cancels as soon as the drain logs that its bound expired and it is probing the backlogged pool (checking the thread pool for a backlog), so the cancel always lands in the probe wait, with no timers involved. The runner's new Output drain for process {Pid} was interrupted. debug log records, inline and with no pool worker, that the interruption was reported while the pool was still held. The backlog is released only after that, because the bounded cleanup afterwards is timer-driven.
    • The deadline is the runner's own timer, which a starved pool can delay, so that case releases the backlog once the token fires and checks only the classification.
    • It fails on main and on this PR's first revision. It also fails deterministically (5/5) with a probe wait that ignores cancellation.
    • It passes 10/10 locally, plus 4/4 under full CPU load.
  • Real-pipe repro on Linux: GitCommandRunner.RunProcessAsync on a real /bin/sh printf, while the pool is flooded.
    • Before the fix: 6/10 RepositoryUnavailable (drain TIMEOUT at about 2.0s, capture completes right after cancel).
    • After the fix: 0/10.
  • Related suites after rebasing on main (ProcessCaptureRunnerTests, ConfigSourceServiceGitTests, GitCommandRunnerTests, PeerInstallProbeTests, GitClientTests, ConfigurationHelperTests): 279 passed, 1 skipped (Windows-only). A 20× stress of the regression test plus RunProcess_DescendantHoldingOutputCannotBlockShutdown passed 20/20.
  • Real CLI: a locally built aspire update --config-source --non-interactive --yes against a real local git config source exited 0 with "configuration updated" and wrote appliedVersion and the docs settings.

Fixes #21625

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
    • No

🤖 Posted by AspireAgent (AI Agent)

@github-actions

github-actions Bot commented Nov 1, 2026

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21626

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21626"

@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Critical issues leave pool-probe waits unbounded by cancellation or deadlines and allow cancellation or timeout during extended draining to be reported as success.

2 open issues
What changed in this PR

This PR addresses intermittent CLI Git-check failures caused by discarding completed process output when thread-pool backlog delays capture continuations.

Changes:

  • Measures thread-pool scheduling delay before abandoning post-exit output capture.
  • Adds a regression test that deliberately backlogs the thread pool.
  • Updates the configuration-source specification to describe backlog-aware draining.
File Description
tests/​Aspire.Cli.Tests/​Utils/​ProcessCaptureRunnerTests.cs Adds an isolated thread-pool backlog regression test.
src/​Aspire.Cli/​Utils/​ProcessCaptureRunner.cs Extends post-exit capture waits when scheduling is delayed.
docs/​specs/​cli-config-sources.md Documents backlog-aware output draining.

Review effort: Balanced

@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The pool-recovery branch can discard pending output capture and incorrectly report RepositoryUnavailable for a completed Git process; it needs a cancellation-aware bounded drain and regression coverage around the initial drain deadline.

No open issues 🎉

1 previously missed issue

In code that hasn't changed since last review

Medium severity Recovery probe can discard output before capture finishes

src/​Aspire.Cli/​Utils/​ProcessCaptureRunner.cs:328

If the pool recovers just before the probe is queued, the probe can finish promptly while GitCommandRunner.ReadOutputAsync is still draining buffered output or awaiting its next read; starting earlier does not guarantee that capture finishes first. This branch then discards the capture and cancels its reads, so a completed Git process can still produce RepositoryUnavailable without a descendant holding the pipe. Allow a fresh, cancellation-aware bounded drain after the pool becomes responsive, and add a regression covering recovery around the initial drain deadline.

2 resolved since last review

Review effort: Balanced

Copy link
Copy Markdown
Collaborator Author

Windows Cli (attempt 1, job 124721834587) failed with one unrelated test:

GitCommandRunnerTests.ConfigureUnattendedSsh_VerifiesOpenSshAndAppliesBatchModeAfterUrlRewrites(rewritten: False, setting: "GIT_SSH_COMMAND"): Unattended Git requires a verified OpenSSH executable…, after 5.2s.

  • The two preceding git calls finished in 0.04s and 0.13s. The remaining ~5s matches the 5s timeout of the ssh -V probe. The runner heartbeat at that moment showed CPU at 86.5%, with git at 99% CPU.
  • This PR can't turn a successful probe into this failure. The first 2s post-exit wait is unchanged. Previously an expired bound always returned a null capture, which fails this check. Now the runner either recovers the complete capture or fails at the same 5s deadline.
  • The test passes locally with this change (GitCommandRunnerTests is in the 279 passed, 1 skipped run).

This is a load-related probe timeout, so I'm rerunning the failed job once.

🤖 Posted by AspireAgent (AI Agent)

Copy link
Copy Markdown
Collaborator Author

Update: CI attempt 1 on d493490 also failed macOS Cli, in this PR's own new assertion. RunAsync_InterruptedDuringBackloggedDrainReportsInterruption(cancelByCaller: True) returned only after the backlog was released.

Cause: in each timer-queue pass .NET fires only the first expired timer on the timer thread and queues the rest to the thread pool. The test's 4s CancelAfter coincided with the drain's second 2s bound, so the cancellation itself could wait behind the test's backlog.

Fix in 44956ed:

  • The interruption test no longer depends on timer delivery. The capture stays open until the token fires, like a reader returning buffered output on cancellation, and the backlog is released on that token. The outcome is deterministic whichever timer fires first.
  • The release-time assertion is removed.
  • The first post-exit wait now observes the token too, so an interruption is reported in every drain phase.
  • Before/after: the theory fails on both main and dac5c31 (Expected TimedOut / Actual null, Expected True / Actual False) and passes 5/5 locally now. Related suites: 278 passed, 1 skipped, 1 unrelated failure seen once (4ms PeerInstallProbeTests start-up, 0 in 15 re-runs).

The new push restarts CI, so the Windows ConfigureUnattendedSsh… rerun described above is superseded. If it recurs I'll handle it as a separate flake.

🤖 Posted by AspireAgent (AI Agent)

@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Pool recovery just after the initial bound can cause valid capture output to be discarded, and the interruption test does not verify completion before pool recovery.

1 open issue
1 previously missed issue

In code that hasn't changed since last review

Medium severity Allow backlog capture to drain after pool recovery

src/​Aspire.Cli/​Utils/​ProcessCaptureRunner.cs:334

If the pool recovers just after the 2s bound, the probe can finish on one worker in under 250ms while another worker is still completing the previously starved capture. This branch then discards the output and cancels the reads even though the delay came from the backlog, not an inherited pipe. Give the capture a bounded drain opportunity after the pool becomes responsive, and cover recovery just after the initial bound.

Review effort: Balanced

Copy link
Copy Markdown
Collaborator Author

Rerunning one unrelated failure in run 41266068726: VS Code extension E2E (Windows, resource-debug-tools-negative) (job).

The test cancels through the VS Code invocation token and reports a stopped resource without invoking a debugger hit Timed out after 10000ms waiting for E2E control revision while the extension was polling aspire resource stop/aspire ps. That path doesn't use ProcessCaptureRunner, which is used only for git, the peer-install probe and configuration helper captures, so this PR doesn't touch it. All the Cli test jobs passed on ubuntu, windows and macOS. Rerunning the failed job once.

🤖 Posted by AspireAgent (AI Agent)

Copy link
Copy Markdown
Collaborator Author

Update on run 41266068726: VS Code extension E2E (Windows, debug-startup-timeout) hung in Run extension E2E tests. It ran about 30 minutes with no progress; the shard usually takes about 6 minutes.

The same shard hung the same way on main in run 41234363151 (job 124653890239, ~47 min, no result), so this isn't caused by this PR. Every other check passed, including all the Cli test jobs.

I cancelled the run and am rerunning only its failed or cancelled jobs once: this shard and the resource-debug-tools-negative shard from my previous comment.

🤖 Posted by AspireAgent (AI Agent)

@aspire-repo-bot
aspire-repo-bot Bot requested a balanced review from Copilot November 1, 2026 04:20
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed to evaluate a cancellation race that can report partial output as successful and a timing-sensitive test that may fail under Windows CI load.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copy link
Copy Markdown
Collaborator Author

On 4855109, the new assertion failed once on ubuntu Cli (job): (cancelByCaller: True, captureObservesCancellation: True) completed only once the backlog was released, after 20s.

The cause was in the test, not the drain. The capture started the backlog after its own WaitForExitAsync plus Task.Yield. On Unix, the runner's own WaitForExitAsync continuation is queued separately to the pool, so it could still be waiting behind the backlog. The runner then sat in the pre-exit wait, which wasn't the path under test.

Fixed in bf6ddb6:

  • The runner logs Process {Pid} exited; draining its output. at debug level when it enters the drain. The test starts the backlog from that log callback, on the runner's own thread.
  • The capture that ignores cancellation explicitly queues behind the backlog on the global queue.

Results: 15/15 locally, 6/6 under full CPU load, and the full Cli suite. A probe wait that ignores cancellation still fails both caller cases at 20s.

🤖 Posted by AspireAgent (AI Agent)

@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because timing-dependent synchronization may let the regression test pass without exercising the intended post-exit drain behavior.

No open issues 🎉

Review effort: Balanced

@aspire-repo-bot
aspire-repo-bot Bot requested a balanced review from Copilot November 1, 2026 05:37
…rruption test deterministic

Timer callbacks after the first in a timer-queue pass are queued to the thread
pool, so a timer-driven cancellation can wait behind the test's backlog. The
interruption test now holds the capture open until the token fires and releases
the backlog on that token, so its outcome no longer depends on timer delivery.
The first post-exit wait also observes the token, so an interruption is reported
in every phase of the drain.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Hold the backlog independently of the operation token for caller
cancellation, cancel from a dedicated thread, and record completion inline
so the test fails if any post-exit wait ignores cancellation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Starting the backlog from the capture could leave the runner's own exit
continuation queued behind it, so the runner sat in the pre-exit wait until
the pool recovered. Log drain entry and start the backlog from that hook.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Log when the post-exit drain is interrupted and when it starts probing the
thread pool. The caller-cancel variants now cancel from a dedicated thread
once the drain is probing the backlogged pool, and assert that the runner
reports the interruption while the backlog is still held, instead of
depending on timers that a starved pool can delay.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@davidfowl
David Fowler (davidfowl) force-pushed the davidfowl/configsource-git-drain-starvation branch from f022a04 to 9aa4922 Compare November 3, 2026 03:12

David Fowler (davidfowl) commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator Author

Rebased onto current main to clear the conflict. The only conflict was the Git-acquisition paragraph in docs/specs/cli-config-sources.md: I kept main's rewritten text (symlink/size-limit changes from #21682) and re-added this PR's two sentences about the post-exit drain extension and its timeout/cancellation bound. The code and test commits applied unchanged. Locally, the ProcessCaptureRunner, ConfigSourceServiceGit, GitCommandRunner, PeerInstallProbe, GitClient and ConfigurationHelper tests pass (279 passed, 1 Windows-only skip).

Update: after the rebase, macOS CI failed RunAsync_InterruptedDuringBackloggedDrainReportsInterruption(cancelByCaller: True, captureObservesCancellation: True) because the interruption was only reported once the backlog released at 20s. The backlog can also starve the 2s drain bound's own timer, so the probe that triggers the caller's cancellation never ran. bb6f406 arms a dedicated-thread fallback at drain entry, so the cancel then lands in the bound's wait. The assertion is unchanged, and that path still has to report the interruption while the pool is held (verified locally by forcing it). This is on top of Mitch Denny (@mitchdenny)'s 0baa442.

🤖 Posted by AspireAgent (AI Agent)

@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The timeout recovery can discard fully written Git output while queued reads are still draining, incorrectly reporting RepositoryUnavailable.

1 open issue

Review effort: Balanced

Give output reads one final bounded, cancellation-aware drain after a responsive pool probe. Cover multi-step capture completion and cancellation during recovery.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because renewed thread-pool backlog during recovery draining can cause completed output to be discarded and repository unavailability to be reported before the operation deadline.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@github-actions

This comment has been minimized.

…d's timer

On macOS CI the backlog also starved the 2s drain bound's timer, so the probe
log that triggers the caller's cancellation only ran once the backlog was
released. Arm a dedicated-thread fallback at drain entry; if the probe is not
reached, the cancel lands in the bound's wait, which must observe it without
the pool just the same.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

Tests selector

2 / 100 PR test projects · 4 PR jobs, from 2 changed files.

Selected PR test projects (2 / 100)

Aspire.Cli.EndToEnd.Tests, Aspire.Cli.Tests

Selected PR jobs (4)

cli-starter-validation, extension-e2e, polyglot, typescript-api-compat


How these were chosen — grouped by what changed

📦 affected project Aspire.Cli
→ 1 test: Aspire.Cli.EndToEnd.Tests

🧪 tests/Aspire.Cli.Tests/Utils/ProcessCaptureRunnerTests.cs (changed test)
→ 1 directly: Aspire.Cli.Tests

Job reasons

Job Triggered by
cli-starter-validation affected project Aspire.Cli
extension-e2e • src/Aspire.Cli/Utils/ProcessCaptureRunner.cs
• affected project Aspire.Cli
polyglot affected project Aspire.Cli
typescript-api-compat affected project Aspire.Cli

Selection computed for commit bb6f406.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed to evaluate a critical cancellation race that can report success with partial output instead of cancellation or timeout.

No open issues 🎉

Review effort: Balanced

@mitchdenny Mitch Denny (mitchdenny) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the recovery-drain fix and subsequent test cancellation fallback. No remaining code findings. CI currently has failures in Windows ARM64 starter validation and TypeScript/Blazor gateway E2E; those remain merge-validation items.

@davidfowl
David Fowler (davidfowl) merged commit 18bc9f6 into main Nov 3, 2026
419 of 424 checks passed
@microsoft-github-policy-service microsoft-github-policy-service Bot added this to the 17.0 milestone Nov 3, 2026
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

✅ No documentation update needed.

Proven false positive → no docs needed (internal bug fix).

  • Triggered signal: pr_body_has_cli_flag_mention. Evidence: the PR body mentions aspire update --config-source --non-interactive --yes only as a manual verification run, not as a new or changed flag.
  • The change is internal to ProcessCaptureRunner: it stops discarding completed process output when the thread pool is backlogged after the process exits. It restores the intended behavior (a transient Windows CI flake in a Git config-source check). No new options, strings, or public surface.
  • The implementation spec docs/specs/cli-config-sources.md was updated in the source repo and is not part of aspire.dev.

David Fowler (davidfowl) added a commit that referenced this pull request Nov 3, 2026
Resolve ProcessCaptureRunner conflict: keep main's thread-pool-aware
post-exit drain (#21626) and, when it genuinely gives up, cancel the reads
and keep the partial capture flagged OutputDrainTimedOut.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-cli ci-health Fixes CI flakiness or CI infrastructure (AspireAgent queue)

Projects

None yet

3 participants