Repository navigation
[release/17.0] Capture CLI diagnostics when an E2E command hangs - #22045
Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
Mitch Denny (mitchdenny) wants to merge 1 commit into
Conversation
## Description This is a diagnostics-only change to the CLI E2E test infrastructure. It doesn't change product code. **Problem.** `SecretDotNetAppHostTests.SecretDeploymentEnvironmentLoadsSelectedAspireSecretsFile` failed twice in 284 PR jobs over 7 days, on unrelated branches (#21650). Both times, `aspire secret set` produced no output for 60s. Those failures can't be root-caused because the artifacts contain no CLI logs. On failure, `TerminalRun.DisposeAsync` types the `~/.aspire/logs` copy command into the terminal. When the foreground CLI command is hung, that text queues behind it and never runs. The 15s prompt wait then times out and `.aspire-diagnostics/` is never created. This gap affects every hung CLI E2E command, not just `secret set`. **Change.** - When the normal capture fails, `TerminalRun` calls a new `SuspendHungCommandAndCaptureDiagnosticsAsync` helper. The helper: 1. Sends Ctrl+Z, which suspends the hung foreground job but keeps its process tree alive. 2. Writes `.aspire-diagnostics/hung-command-processes.txt`, containing `date`, `jobs -l` and `ps -eo pid,ppid,pgid,stat,etime,args`. If procps is missing, it falls back to reading `/proc`. 3. Runs the existing capture of `~/.aspire/logs`, packages and DCP logs. 4. Kills every suspended job's whole process group, including pipeline members, so the terminal can exit. - Completion is detected with a unique split marker, because the prompt counter is out of sync after a hang. - The success path is unchanged. The CLI file logger writes at Debug level with `AutoFlush`, so the next occurrence will show which phase (`msbuild -getProperty`, user-secrets init, and so on) and which child process stalled. Contributes to #21650. The issue stays open until the next occurrence is root-caused. ### Validation New `HungCommandDiagnosticsTests` drives a real pty bash through Hex1b. It runs a pipeline that never returns, then asserts three things: - The process snapshot contains both stopped pipeline members. - A `~/.aspire/logs` file is copied. - The shell becomes usable again, with no pipeline member left running. Results: - **With the fix:** passed every local run, about 1s each. The snapshot shows `[1]+ Stopped ...` and `sleep 3601` with STAT `T` and its elapsed time. - **Without the fix** (the Ctrl+Z removed, which reproduces the current behavior): fails after 15s with `WaitUntil timed out ... waiting for: text "hung-command-recovered-..."`. That is the same mechanism that lost the diagnostics in the CI failures. - **With a leader-only kill** (`kill $(jobs -p)`): fails with `WaitUntil timed out ... sleep-processes:0`, because the non-leader pipeline member survives. ## Checklist - Is this feature complete? - [x] Yes. Ready to ship. - [ ] No. Follow-up changes expected. - Are you including unit tests for the changes and scenario tests if relevant? - [x] Yes - [ ] No - Did you add public API? - [ ] Yes - [x] No - Does the change make any security assumptions or guarantees? - [ ] Yes - [x] No **🤖 Posted by AspireAgent (AI Agent)** --------- Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> (cherry picked from commit 73f8459)
14 tasks
This comment has been minimized.
This comment has been minimized.
Mitch Denny (mitchdenny)
marked this pull request as ready for review
November 10, 2026 05:13
Member
Author
|
Approved for inclusion in release/17.0. Do not merge yet. |
Mitch Denny (mitchdenny)
requested review from
James Newton-King (JamesNK),
Adam Ratzman (adamint),
David Negstad (danegsta),
David Fowler (davidfowl),
Eric Erhardt (eerhardt),
Ella Hathaway (ellahathaway),
Karol Zadora-Przylecki (karolz-ms),
Ankit Jain (radical) and
Sébastien Ros (sebastienros)
as code owners
November 10, 2026 06:15
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-47-pr21651
branch
from
November 10, 2026 06:17
73aea9f to
a313b0a
Compare
Contributor
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 22045Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 22045" |
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-47-pr21651
branch
from
November 10, 2026 06:20
a313b0a to
73aea9f
Compare
This comment has been minimized.
This comment has been minimized.
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-47-pr21651
branch
from
November 10, 2026 08:01
73aea9f to
a313b0a
Compare
Mitch Denny (mitchdenny)
force-pushed
the
mitchdenny-backport-17-0-47-pr21651
branch
from
November 10, 2026 08:30
a313b0a to
73aea9f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Backport of #21651 to
release/17.0(cherry-picked from 73f8459 with no modifications).Layer 47 of 55 in the release/17.0 backport stack; stacked on #22044.
Backport recommendation: No — Diagnostics only.
Validation: Stable patch IDs match (
fd56d35972bdeab396b2588e7c79b6ee20ddae64). Builds/tests not run for this unchanged cherry-pick.Backport of #21651
Checklist
<remarks />and<code />elements on your triple slash comments?