Skip to content

[release/17.0] Capture CLI diagnostics when an E2E command hangs - #22045

Closed
Mitch Denny (mitchdenny) wants to merge 1 commit into
mitchdenny-backport-17-0-46-pr21013from
mitchdenny-backport-17-0-47-pr21651
Closed

Mitch Denny (mitchdenny) wants to merge 1 commit into
mitchdenny-backport-17-0-46-pr21013from
mitchdenny-backport-17-0-47-pr21651

Conversation

@mitchdenny

Copy link
Copy Markdown
Member

Description

Backport of #21651 to release/17.0 (cherry-picked from 73f8459 with no modifications).

Layer 47 of 55 in the release/17.0 backport stack; stacked on #22044.

Backport recommendation: No — Diagnostics only.

Validation: Stable patch IDs match (fd56d35972bdeab396b2588e7c79b6ee20ddae64). Builds/tests not run for this unchanged cherry-pick.

Backport of #21651

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

## Description

This is a diagnostics-only change to the CLI E2E test infrastructure. It
doesn't change product code.

**Problem.**
`SecretDotNetAppHostTests.SecretDeploymentEnvironmentLoadsSelectedAspireSecretsFile`
failed twice in 284 PR jobs over 7 days, on unrelated branches (#21650).
Both times, `aspire secret set` produced no output for 60s. Those
failures can't be root-caused because the artifacts contain no CLI logs.

On failure, `TerminalRun.DisposeAsync` types the `~/.aspire/logs` copy
command into the terminal. When the foreground CLI command is hung, that
text queues behind it and never runs. The 15s prompt wait then times out
and `.aspire-diagnostics/` is never created. This gap affects every hung
CLI E2E command, not just `secret set`.

**Change.**
- When the normal capture fails, `TerminalRun` calls a new
`SuspendHungCommandAndCaptureDiagnosticsAsync` helper. The helper:
1. Sends Ctrl+Z, which suspends the hung foreground job but keeps its
process tree alive.
2. Writes `.aspire-diagnostics/hung-command-processes.txt`, containing
`date`, `jobs -l` and `ps -eo pid,ppid,pgid,stat,etime,args`. If procps
is missing, it falls back to reading `/proc`.
3. Runs the existing capture of `~/.aspire/logs`, packages and DCP logs.
4. Kills every suspended job's whole process group, including pipeline
members, so the terminal can exit.
- Completion is detected with a unique split marker, because the prompt
counter is out of sync after a hang.
- The success path is unchanged.

The CLI file logger writes at Debug level with `AutoFlush`, so the next
occurrence will show which phase (`msbuild -getProperty`, user-secrets
init, and so on) and which child process stalled.

Contributes to #21650. The issue stays open until the next occurrence is
root-caused.

### Validation

New `HungCommandDiagnosticsTests` drives a real pty bash through Hex1b.
It runs a pipeline that never returns, then asserts three things:
- The process snapshot contains both stopped pipeline members.
- A `~/.aspire/logs` file is copied.
- The shell becomes usable again, with no pipeline member left running.

Results:
- **With the fix:** passed every local run, about 1s each. The snapshot
shows `[1]+ Stopped ...` and `sleep 3601` with STAT `T` and its elapsed
time.
- **Without the fix** (the Ctrl+Z removed, which reproduces the current
behavior): fails after 15s with `WaitUntil timed out ... waiting for:
text "hung-command-recovered-..."`. That is the same mechanism that lost
the diagnostics in the CI failures.
- **With a leader-only kill** (`kill $(jobs -p)`): fails with `WaitUntil
timed out ... sleep-processes:0`, because the non-leader pipeline member
survives.

## Checklist

- Is this feature complete?
  - [x] Yes. Ready to ship.
  - [ ] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
  - [x] Yes
  - [ ] No
- Did you add public API?
  - [ ] Yes
  - [x] No
- Does the change make any security assumptions or guarantees?
  - [ ] Yes
  - [x] No

**🤖 Posted by AspireAgent (AI Agent)**

---------

Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit 73f8459)
@github-actions

This comment has been minimized.

@mitchdenny
Mitch Denny (mitchdenny) marked this pull request as ready for review November 10, 2026 05:13
@mitchdenny

Copy link
Copy Markdown
Member Author

Approved for inclusion in release/17.0. Do not merge yet.

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 22045

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 22045"

@mitchdenny
Mitch Denny (mitchdenny) force-pushed the mitchdenny-backport-17-0-47-pr21651 branch from a313b0a to 73aea9f Compare November 10, 2026 06:20
@github-actions

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants