Skip to content

Bound primary apt mirror builds before CLI E2E fallback - #21849

Open
David Pine (IEvangelist) wants to merge 4 commits into
mainfrom
ievangelist-bound-ci-mirror-fallback
Open

David Pine (IEvangelist) wants to merge 4 commits into
mainfrom
ievangelist-bound-ci-mirror-fallback

Conversation

@IEvangelist

@IEvangelist David Pine (IEvangelist) commented Nov 6, 2026 •

Copy link
Copy Markdown
Member

Description

Shared CLI E2E image builds can exhaust their 45-minute job downloading from a throttled Azure Ubuntu apt mirror without reaching the existing error-only fallback. Bound primary attempts so default-source fallback also runs for slow or stalled builds.

Independent fix based on main; relates to #21828, which is unchanged and is not a dependency. No auto-merge or bypass requested.

  • Share orchestration across the reusable image workflow and cache warmer. Apply a five-minute primary limit to .NET, polyglot base, Java and Go. Four primary attempts plus four 30-second cleanup grace periods leave 23 minutes within the unchanged 45-minute cap for fallback, setup and artifact handling.
  • TERM the client's process group so Docker can cancel its BuildKit session. Retry exactly once after ordinary failure or deadline, preserving both attempts' output and fallback status. A dependency-free Node supervisor tracks deadline and external signals independently, including cancellation during cleanup. External cancellation and forced kills, including an independently killed client, fail without fallback.
  • Preserve original Ubuntu apt sources and restore them for empty mirror arguments, including inherited Java/Go layers. Leave third-party sources untouched.
  • Add behavioral regressions, test-selection routes, cache-warmer filters and docs. No action pins, package manifests or product code changed.

Two-attempt root-cause evidence

Both jobs ran at #21828 head 1ab3ad1e302a110bdb4f69ad4e3ee2e05d646d9a in run 42151619597. Both have annotation The job has exceeded the maximum execution time of 45m0s.

Attempt Image job Started / completed (UTC, November 6) Exact apt log
1 127676748230 02:41:00 / 03:26:22 #11 2272.6 Fetched 82.4 MB in 37min 49s (36.3 kB/s)
2 127692798471 03:29:03 / 04:14:24 #11 2492.6 Fetched 108 MB in 41min 29s (43.4 kB/s)

Attempt 2 then downloaded from HTTPS GitHub at #11 2497.2 Fetched 15.1 MB in 0s (80.1 MB/s), while another mirror download logged #15 92.37 Fetched 4031 kB in 1min 29s (45.4 kB/s). Both also reported GHA cache-import HTTP 400, Our services aren't available right now. Cold apt downloads, not product tests, consumed the image job. Neither attempt reached product-test execution using these images. No third rerun was triggered.

Local validation

.\restore.cmd: passed, 0 errors and 18 dependency-advisory warnings.

Original selector coverage passed across appropriate hosts: 299 Windows managed cases, 2 Windows native MSBuild graph cases and 79 Linux shell/Git cases:

dotnet test --project tests\Infrastructure.Tests\Infrastructure.Tests.csproj --no-launch-profile -- --filter-namespace "Infrastructure.Tests.TestTriggerMap" --filter-not-class "*.PrComparisonBaseTests" --filter-not-class "*.SelectTestsCliTests" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
# Passed: 299

dotnet test --project tests\Infrastructure.Tests\Infrastructure.Tests.csproj --no-launch-profile -- --filter-method "*.NativeDashboardSharedHelpersSelectNativeValidationInEnforceMode" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
# Passed: 2

dotnet test --project tests\Infrastructure.Tests\Infrastructure.Tests.csproj --no-launch-profile -- --filter-class "*.TestTriggerMapTests" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true"
# Passed: 179 after the final routing change

dotnet run --project tools\SelectTests -- --changed-files <generated-changed-files.txt> --explain
# Passed: production graph/prefilter, 0 unattributed inputs.
# Shared Docker shell helper correctly activates the existing ALL route.

Linux container, pinned .NET runtime, same built MTP assembly:

/tmp/dotnet/dotnet /repo/artifacts/bin/Infrastructure.Tests/Debug/net11.0/Infrastructure.Tests.dll --filter-class "*.PrComparisonBaseTests" --filter-class "*.CliE2EImageBuildTests" --filter-class "*.SelectTestsCliTests" --filter-not-method "*.NativeDashboardSharedHelpersSelectNativeValidationInEnforceMode" --filter-not-trait "quarantined=true" --filter-not-trait "outerloop=true" --results-directory /tmp/test-results
# Original implementation: passed 88 (79 selector + 9 image cases).

/tmp/dotnet/dotnet artifacts/bin/Infrastructure.Tests/Debug/net11.0/Infrastructure.Tests.dll --filter-class "*.CliE2EImageBuildTests" --filter-not-trait quarantined=true --filter-not-trait outerloop=true
# Final b8274fc8e0: passed all 13 expanded image cases.

The initial Windows run failed in 17 shell cases because it selected the broken System32 WSL bash shim, and 5 existing Git-fixture cleanup cases with UnauthorizedAccessException. The same cases passed on Linux. The 2 native graph cases passed on Windows; running the Windows SDK on Linux caused BadImageFormatException. Passing partitions cover every focused case without modifying unrelated tests.

The 13 image cases cover success, ordinary failure, fallback failure, slow success, stall, forced kill, client cancellation, external cancellation before/during deadline cleanup, client SIGKILL during cleanup, all-image/.NET-only orchestration, and exact original apt-source restoration in real Ubuntu. Cleanup cancellation reproduces on 4bb054c as incorrect exit 0 plus fallback; the corrected supervisor exits 143 without fallback. Killed clients retain 137, never deadline 124.

Real Docker/Buildx probe re-run after the supervisor change:

build_with_mirror_retry 20s "real BuildKit cancellation probe" docker buildx build --progress plain --load -t mirror-fallback-probe

Passed: primary printed PRIMARY_RUNNING, then #5 CANCELED / Canceled: context canceled; fallback printed FALLBACK_RUNNING and loaded the image. BuildKit history recorded primary Error and fallback Completed. Process inspection before fallback and afterwards found only the BuildKit daemon, no surviving primary RUN/sleep. Temporary builder and image removed. git diff --check: passed.

Actual final-head CI

Head b8274fc, run 42170803717: image build and all four uploads passed; Infrastructure passed 2090/2090; all 116 selected CLI archive E2E jobs passed. The downloaded Infrastructure TRX confirms all 13 new image cases passed. CLI unit tests passed on Linux, macOS and Windows. This is actual final-head evidence, not evidence from superseded runs.

Image job 127739541935 ran 05:56:03–06:05:32 UTC. All four artifacts are present and unexpired:

Variant Artifact ID Size (bytes) Uploaded UTC
.NET 13481517063 696399767 06:04:42
Polyglot 13480867983 391644472 06:04:54
Java 13481671757 654944092 06:05:13
Go 13481043964 583388699 06:05:30

The primary mirror was healthy: #11 4.388 Fetched 108 MB in 1s (114 MB/s). No actual CI timeout/fallback occurred; fallback evidence is deterministic/local BuildKit evidence, not a production CI claim.

Representative actual consumers confirm all four artifacts load and execute tests:

Consumer job Images loaded Test outcome
Go starter 127743182589 .NET, polyglot, Go 1 passed, 1 skipped
Java polyglot 127743185239 .NET, polyglot, Java 1 passed
.NET smoke 127743188570 .NET, polyglot 10 passed

Required checks are not green; approval is still required. Two failures outside the changed image infrastructure remain, with same-source main controls rather than blind reruns:

  • Windows extension units 127739542026: 4150 passed, 9 pending, 1 failed. rejects when a POSIX process group remains alive after forced termination passed its rejection/listener assertions but saw global fake-timer count 2 instead of 0. Test and process-helper blobs are identical to main 54b4929. Main control 127718292677 passed that exact case and all 4151 tests. The source of the two timers is not established; no extension edit or retry made.
  • Windows Hosting-2 127739572340: ExecutableDoesNotExist and ExecutableWorkingDirectoryDoesNotExist failed before scenario assertions because DCP's docker.exe --version dependency probe timed out. Test file and entire DCP source tree are identical to main 54b4929. Main control 127718297752 TRX passed the same cases in 6.005s and 2.854s. No product edit or retry made.

All four review threads have been addressed. No merge or bypass performed; #21828 remains untouched and cannot consume this fix until integration.

Security considerations

The supervisor executes existing checked-in Docker commands, not new untrusted input. Separate-session process-group signaling avoids signaling the workflow shell and permits child cleanup. A forced kill cannot confirm remote BuildKit cancellation, so it fails closed without fallback. No threat model or separate security review has been performed.

Fixes # (issue)

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

Move shared image orchestration into a testable shell entry point, reserve fallback time inside the existing job budget, and fail closed on unsafe cancellation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21849

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21849"

@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Derived-image fallbacks can reuse the throttled Azure mirror and exhaust the job, while external cancellation can incorrectly trigger a retry that reports success.

2 open issues
What changed in this PR

This PR bounds primary apt-mirror attempts in shared CLI E2E image builds, addressing the slow downloads that blocked validation in #21828 without depending on that PR.

Changes:

  • Extracts image-build orchestration into a shared script with five-minute primary limits and fallback handling.
  • Adds behavioral coverage for retries, cancellation, cleanup, and image selection.
  • Updates test routing, cache-warmer triggers, and documentation.
File Description
tests/​Infrastructure.Tests/​TestTriggerMap/​TestTriggerMapTests.cs Covers test selection for the shared script.
tests/​Infrastructure.Tests/​Pipelines/​CliE2EImageBuildTests.cs Registers nine Linux behavioral scenarios.
tests/​Infrastructure.Tests/​Pipelines/​cli-e2e-image-build.harness.ts Exercises orchestration, retries, and process cleanup.
eng/​scripts/​build-cli-e2e-images.sh Centralizes builds with bounded primary attempts.
eng/​github-ci/​test-trigger-map.yml Routes script changes to E2E and infrastructure tests.
docs/​ci/​cli-e2e-images.md Documents timeout budgets and fallback behavior.
.github/​workflows/​warm-cli-e2e-image-cache.yml Triggers cache warming when the script changes.
.github/​workflows/​build-cli-e2e-image.yml Replaces inline orchestration with the shared script.

Review effort: Balanced

Comment thread eng/scripts/build-cli-e2e-images.sh Resolved
Comment thread eng/scripts/build-cli-e2e-images.sh Resolved
Snapshot original Ubuntu sources before configuring mirrors so derived-image fallbacks restore defaults. Normalize cancellation in a supervising shell even when Docker exits with code 1 after TERM. Add external-signal and real Ubuntu restoration regressions.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

An external TERM during deadline cleanup can be ignored, allowing fallback to report success after cancellation; cancellation must be tracked independently and covered by a regression test.

1 open issue
2 resolved since last review

Review effort: Balanced

Comment thread eng/scripts/build-cli-e2e-images.sh Resolved
Track primary deadlines and external signals independently; retain process-group TERM and fail-closed kill grace.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A SIGKILL during deadline cleanup can be misreported as exit code 124, allowing fallback to report success without confirmed graceful BuildKit cancellation.

1 open issue
1 resolved since last review

Review effort: Balanced

Comment thread eng/scripts/run-bounded-image-build.ts Resolved
@github-actions

This comment has been minimized.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Tests selector

Selects the full PR test matrix + all PR-gated jobs (ALL) — a rule matching 'tests/Shared/Docker/configure-ubuntu-apt-mirror.sh' selects ALL


Selection computed for commit b8274fc.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed to evaluate a critical timeout-cleanup risk that could allow fallback to overlap a surviving primary process.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

David Pine (IEvangelist) added a commit that referenced this pull request Nov 6, 2026
## Description

Go starters should build from their shipped files without an initial
`aspire run` repairing dependencies, and Go preview validation should
exercise the same runtime and native-runner paths as other AppHosts.

- Ship a tidy API `go.mod` and generated `go.sum`. Add starter scenarios
checking gitignore, readonly API build, tidy stability, health and
weather responses; remove the Kubernetes manual-tidy workaround.
- Honor the existing `--use-redis-cache false` option for Go.
Omitted/true retains Redis; false omits the integration and resource.
Cover all three scaffold choices with generated-source snapshots and
both HTTP lifecycle variants with Docker E2E.
- Add `go vet` to Go polyglot fixtures, an EF Core fixture, and Foundry
tool/AKS node-pool parity. Exercise configuration-section list getters
in the real Go runtime smoke.
- Add Go to all six native starter-validation platforms, with official
checksum-verified Go installation, timeout budget, and runtime-only
selector edges for Go hosting, Go codegen, and Redis. Linux retains
API/Redis readiness; containerless Windows/macOS use the cache-free
starter with API readiness unchanged.
- Add isolated-local-feed Go/Java scaffolding and restore to the
stabilization smoke without rewriting template pins. Use a genuine
Release bundled driver outside repository mode, cold consumer caches,
and exact hosting-package extraction/provenance checks. Enable Java only
in its subprocess.
- Select languages by name rather than picker position. Restore stale
Java/TypeScript SDK baselines after `aspire add` so the following
`aspire restore` genuinely tests regeneration. Establish the TypeScript
baseline with an actual restore before reading cache metadata.
- Wait for the TypeScript Kubernetes application's actual Ready pod;
exercise the Go Helm namespace/chart-version callback.

Relates to #21202, #21206, #21209, #21210, #21201, and #21108.

### User-facing usage

```bash
# Default retains Redis.
aspire new aspire-go-starter --name GoReady --non-interactive --suppress-agent-init

# Containerless starter: the Go API runs without Redis or a Docker daemon.
aspire new aspire-go-starter --name GoReady --use-redis-cache false --non-interactive --suppress-agent-init
```

Both variants emit `api/go.sum` and an already-tidy API module. Its API
can build with `go build -mod=readonly` without first running the
AppHost's tidy resource.

### Dependencies and remaining acceptance

This PR is independent of #21823. Root-module brownfield E2E #21825
merged into the #21823 feature branch, not directly into main; its
delivery follows #21823. #21806 independently covers other Go
init/build/vet/stable-restore scenarios.

The existing stabilization gate remains disabled
(`STABILIZATION_ENABLED: 'false'`); enabling it needs a separate
CI/policy decision. Current `Aspire.Hosting.Java.csproj` still sets
`SuppressFinalPackageVersion=true` while the Java starter pins the
requested Aspire version. The corrected smoke reaches actual NuGet
restore and exposes that mismatch: `Aspire.Hosting.Java (= 17.0.0)` is
unavailable, the nearest local package is `17.0.0-dev`, and nuget.org
versions are not considered. No pin substitution or Java release-policy
change is included.

Native Linux validation built the genuine stabilized Release CLI/payload
and proved cold Go/Redis package restore and C# restore. This is **not a
clean stabilization pass**: the unsuppressed stable pack first
reproduced NU5104 on existing .NET 11 preview dependencies. The
diagnostic consumer feed downgraded only NU5104 with
`WarningsNotAsErrors=NU5104`; that exception is not committed. The final
smoke correctly exits 1 with only Java in its failure list.

The bounded shared-image fix is separate in #21849; this PR does not
modify image Dockerfiles or the image-build implementation. Its
`b8274fc8e0` image job and all four uploads passed, Infrastructure
passed 2090/2090 including all 13 new image cases, and all 116 CLI
archive E2E jobs passed. Earlier Windows extension fake-timer and DCP
Docker-startup failures were investigated with same-source passing main
controls; the morning live refresh reports its Final Results passing. It
remains independently review-gated. Do not infer actual timeout/fallback
execution from a healthy-mirror image pass.

No action SHA is added or changed. No official Azure DevOps pipeline is
modified. Parent's Go-minimum-version map rule/theory and the other
owner's extension-e2e map block are untouched. The cache option uses
existing embedded CLI inputs; no additional selector rule is required.
The stabilization script remains advisory/disabled rather than being
presented as a regular-PR gate.

This does not close #21108: live Azure Container Apps deployment and
real missing/old-toolchain E2E remain uncovered here. Cross-language
lifecycle redesigns are deferred.

### CI failures diagnosed and fixed

Original-head run
[42116644663](https://github.com/microsoft/aspire/actions/runs/42116644663)
exposed two causes:

- Windows x64/ARM64 and macOS x64/ARM64 starter validation failed
because Redis could not start: the Windows Docker engine pipe was absent
and macOS had no Docker executable. The API correctly waited on its
unhealthy dependency. Those native runners now use the supported
cache-free variant; Linux and Docker E2E retain Redis coverage. No
readiness assertion was removed.
- `TypeScriptCodegenValidationTests` read `.codegen-hash` immediately
after init, which supplies SDK sources but not restore-cache metadata.
An explicit baseline restore now precedes the snapshot and stale-cache
assertions.

**Head `1ab3ad1e30`: all six native starter lanes passed**, including
both Redis-enabled Ubuntu lanes and all four formerly failing
Windows/macOS lanes, in [run
42151619597](https://github.com/microsoft/aspire/actions/runs/42151619597).
Packaged CI subsequently passed
[GoStarterReadinessTests](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127721672515)
(both cache choices),
[GoStarterTemplateTests](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127721672968),
[JavaCodegenValidationTests](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127721672216),
and
[TypeScriptCodegenValidationTests](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127721664876).
All originally failing targeted lanes/classes therefore have passing
coverage.

The shared image job first timed out at 45 minutes twice. Attempt 2
fetched 108 MB from Azure Ubuntu in 41 minutes 29 seconds (43.4 kB/s),
versus a following GitHub download at 80.1 MB/s; GHA cache import also
returned HTTP 400 service-unavailable responses. After #21849's
same-input image control passed with all four uploads, one
evidence-based product revalidation was started. [Attempt 3's image job
passed](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127718680817)
and admitted packaged E2E. No blind retries were made. The successful
infrastructure control used a healthy primary mirror; it does not
demonstrate timeout/fallback execution in that CI run.

**Head `bd30794b9e`: historical failures and final outcome are
distinct.** In [run
42167868738](https://github.com/microsoft/aspire/actions/runs/42167868738),
all six native starter lanes and Go/Java/TypeScript SDK validation
passed. The first [shared image
job](https://github.com/microsoft/aspire/actions/runs/42167868738/job/127729647462)
was canceled after 45 minutes 22 seconds with Azure Ubuntu apt downloads
still running and all four uploads skipped; the initial Linux
tree-actions attempt also failed on a stale immutable Running snapshot.
The morning live refresh subsequently shows [Final Results
passing](https://github.com/microsoft/aspire/actions/runs/42167868738/job/127754717945),
including
[GoStarterReadinessTests](https://github.com/microsoft/aspire/actions/runs/42167868738/job/127747615122).
This session did not initiate retries. A passing later attempt does not
repair the tree-actions race: its separate test-only fix #21854 still
needs independent approval/merge and incorporation before claiming the
race fixed in this checkout.

**Latest head `0bb6569d91` requires fresh checks**, now running in
[42222384609](https://github.com/microsoft/aspire/actions/runs/42222384609).
Both new review findings are corrected: SDK selection accepts only exact
numeric MAJOR.MINOR.PATCH package filenames, and Go starter startup is
inside its existing cleanup scope. Both threads were replied to and
resolved with evidence. Earlier-head CI is not presented as latest-head
proof.

Per the user's explicit authorization, the coordinator enabled normal
native **squash auto-merge** at 05:22:46Z on 2026-11-06. It remains
enabled behind required fresh checks and review requirements. No admin
bypass is authorized or used; the PR is not merged.

### Stabilization review corrections

The smoke now publishes a Release CLI with its real stabilized bundle
outside the checkout, propagates stabilization into child payload
builds, and clears inherited `ASPIRE_REPO_ROOT` for every consumer
invocation. Consumer packages are isolated after the driver build so
build-time extraction cannot hide a missing stable package. Successful
guest restores must produce metadata for the exact hosting pins,
including Redis for Go; provenance is retained in the output.

The real run also exposed two stale smoke assumptions: a pre-owned
parent feed mapping conflicts with scaffolding's project-local channel
policy, and current C# file-based templates use `#:package
Aspire.Hosting.AppHost@<version>`, not the obsolete SDK directive. The
smoke now lets the CLI establish its own authoritative project-scoped
feed mapping and checks the current hosting-package contract. No
customer NuGet policy is changed.

### Validation

- `.\restore.cmd` - passed.
- `dotnet build
tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj
--no-restore --nologo -v quiet` - passed, zero warnings/errors; repeated
successfully after removing temporary fault injection for `0bb6569d91`.
- `dotnet test --project tests\Aspire.Cli.Tests\Aspire.Cli.Tests.csproj
--no-launch-profile -- --filter-class
'*.NewCommandTemplateConfigPersistenceTests' --filter-not-trait
'quarantined=true' --filter-not-trait 'outerloop=true'` - passed, 59
tests including default/true/false Go package and generated-source
assertions. Existing ILCompiler NU1903 advisory was emitted.
- With
`ASPIRE_E2E_CLI_ARCHIVE_DIR=artifacts\readiness-ci\cli-archive-new` and
`ASPIRE_E2E_DOTNET_IMAGE=aspire-e2e-polyglot-base:latest`, `dotnet test
--project
tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj
--no-launch-profile -- --filter-method
'*.RestoreGeneratesSdkFiles_WithConfiguredToolchain' --filter-not-trait
'quarantined=true' --filter-not-trait 'outerloop=true'` - passed, all
four npm/bun/yarn/pnpm cases against exact `1ab3ad1e30` Linux
CLI/packages. The corresponding packaged CI class also passed above.
- With
`ASPIRE_E2E_CLI_ARCHIVE_DIR=artifacts\readiness-ci\cli-archive-new` and
`ASPIRE_E2E_POLYGLOT_GO_IMAGE=aspire-go-e2e-6972:latest`, `dotnet test
--project
tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj
--no-launch-profile -- --filter-method
'*.StarterShipsTidyApiAndServesRequests' --filter-not-trait
'quarantined=true' --filter-not-trait 'outerloop=true'` - 1 passed/1
failed locally against exact `1ab3ad1e30`. Cache-free passed actual
scaffolding, readonly API build, tidy check, API-up, health/weather and
stop without a Docker socket. Redis timed out locally. The unchanged
`2859e199e2` control reproduced the same timeout on this Docker Desktop
host; no workaround or weaker assertion was added. **Both cases
subsequently passed packaged CI**, linked above. After the cleanup-scope
correction, this local command again passed cache-free and failed Redis
on API readiness; the independent CLI stop log confirmed successful
shutdown, while the shared terminal helper observed the prior ERR
prompt. That local run is not claimed green.
- Temporary fault-injection validation used that same E2E command with
only the cache-free case and a shell wrapper returning HTTP 503 for the
Dashboard probe after a real packaged AppHost launch - intentionally
failed. Terminal output showed `dashboard-http-failed`, then cleanup
invocation; the CLI log confirmed `apphost.go stopped successfully` and
exit 0. The wrapper and temporary test edit were removed before the
final clean build/commit. This verifies partial-start cleanup, not a
successful readiness scenario.
- `docker run --rm --entrypoint bash -v "${PWD}:/workspace:ro"
aspire-go-e2e-6972:latest -c "sed 's/\r$//'
/workspace/artifacts/readiness-ci/validate-stable-sdk-selection.sh |
bash"` - passed actual smoke-script preflight cases for missing SDK,
prerelease-only, mixed
stable/dev/preview/rc/build-metadata/four-component filenames,
multi-digit stable versions, and feed paths with spaces. The
instrumented driver received `BundleVersion=17.0.0` from the mixed feed.
Invalid-only feeds exited 1 with the explicit missing-stable diagnostic.
Temporary harness removed after retaining logs; not a new all-language
stabilization pass.
- `dotnet test --project
tests\Infrastructure.Tests\Infrastructure.Tests.csproj
--no-launch-profile -- --filter-class '*.TestTriggerMapTests'
--filter-not-trait 'quarantined=true' --filter-not-trait
'outerloop=true'` - passed, 176 tests.
- `dotnet run --project tools\SelectTests -- --changed-files
artifacts\readiness-changed-files.txt --explain` - passed against the
real graph/map; Go hosting/codegen, Redis, starter checksum and EF Core
fixture inputs select starter-validation/polyglot with zero unattributed
files. Schedule-only deployment remains advisory.
- `$env:DOTNET_ROOT = Join-Path $PWD '.dotnet'; $env:ASPIRE_REPO_ROOT =
$PWD; .\artifacts\bin\Aspire.Cli\Debug\net11.0\aspire.exe restore
--apphost tests\PolyglotAppHosts\<integration>\Go\apphost.go
--non-interactive` - passed separately for
`Aspire.Hosting.EntityFrameworkCore`, `Aspire.Hosting.Foundry`, and
`Aspire.Hosting`, generating actual SDK bindings.
- `docker run --rm --entrypoint bash -v "${PWD}:/workspace"
aspire-go-e2e-6972:latest -c 'set -e; for fixture in
Aspire.Hosting.EntityFrameworkCore Aspire.Hosting.Foundry
Aspire.Hosting; do cd "/workspace/tests/PolyglotAppHosts/$fixture/Go";
gofmt -w apphost.go; go build -buildvcs=false -o /tmp/fixture ./...; go
vet ./...; done'` - passed for all three fixtures.
- `docker run --rm --entrypoint bash -v "${PWD}:/workspace:ro"
aspire-go-e2e-6972:latest -c 'set -e; bash -n
/workspace/.github/workflows/polyglot-validation/test-go.sh
/workspace/.github/workflows/polyglot-validation/test-go-playground.sh
/workspace/eng/scripts/stabilization-smoke-init-restore.sh;
work=$(mktemp -d); trap "rm -rf $work" EXIT; cp
/workspace/src/Aspire.Cli/Templating/Templates/go-starter/api/*
"$work/"; cd "$work"; sed -i "s/{{projectNameLower}}/starter/g" go.mod;
go build -mod=readonly -o /tmp/go-ready-api .; go mod tidy -diff; go vet
./...'` - passed without a dependency-repair command.
- Native Linux `./restore.sh -p:SkipTestProjects=true
-p:SkipPlaygroundProjects=true` - passed with the pinned SDK.
- Native Linux `./dotnet.sh pack
src/Aspire.Hosting.AppHost/Aspire.Hosting.AppHost.csproj -c Release
-p:StabilizePackageVersion=true -p:SkipNativeBuild=true --nologo -v
quiet` - failed as expected on existing NU5104 preview dependencies; not
counted as a stabilization pass.
- To isolate consumer behavior, the diagnostic feed used `for project in
Aspire.Hosting Aspire.Hosting.AppHost Aspire.Hosting.Dotnet
Aspire.TypeSystem Aspire.Hosting.CodeGeneration.Go
Aspire.Hosting.CodeGeneration.Java Aspire.Hosting.Go
Aspire.Hosting.Redis Aspire.Hosting.Java Aspire.Hosting.JavaScript
Aspire.AppHost.Sdk Aspire.ProjectTemplates; do ./dotnet.sh pack
"src/$project/$project.csproj" -c Release
-p:StabilizePackageVersion=true -p:SkipNativeBuild=true
-p:WarningsNotAsErrors=NU5104 --nologo -v quiet; done` - passed. No
suppression is committed; Java remains `17.0.0-dev` by existing policy.
- Native Linux `STABILIZATION_SMOKE_CONFIGURATION=Release
ASPIRE_REPO_ROOT=/work/source bash
eng/scripts/stabilization-smoke-init-restore.sh` - built the genuine
stabilized Release driver/payload and completed the consumer flow.
**Exit 1 is the expected Java policy failure, not an all-language
pass.** Go/Redis `17.0.0` metadata names
`/work/source/artifacts/packages/Release/Shipping`; C# `17.0.0`
restored; Java reached real NuGet restore and rejected its absent stable
pin with no nuget.org fallback. Output assertions confirmed the final
failure list is exactly `java`. Sources were `1ab3ad1e30` plus the final
smoke correction; no mixed PR-stamped CLI was used.
- `bash -n eng/scripts/stabilization-smoke-init-restore.sh`, PowerShell
parser validation of `eng\scripts\cli-starter-validation.ps1`, and `git
diff --check` - passed.
- Full `Infrastructure.Tests.TestTriggerMap` namespace was not passed on
this Windows host: WSL's `bash.exe` cannot find `/bin/bash`; a narrowed
run also encountered readonly `.git` cleanup failures. Not counted as
successful validation.

## Checklist

- Is this feature complete?
  - [ ] Yes. Ready to ship.
  - [x] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
  - [x] Yes
  - [ ] No
- Did you add public API?
  - [ ] Yes
    - If yes, did you have an API Review for it?
      - [ ] Yes
      - [ ] No
- Did you add `<remarks />` and `<code />` elements on your triple slash
comments?
      - [ ] Yes
      - [ ] No
  - [x] No
- Does the change make any security assumptions or guarantees?
  - [ ] Yes
    - If yes, have you done a threat model and had a security review?
      - [ ] Yes
      - [ ] No
  - [x] No

---------

Co-authored-by: IEvangelist <7679720+IEvangelist@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Mitch Denny (mitchdenny) pushed a commit that referenced this pull request Nov 10, 2026
## Description

Go starters should build from their shipped files without an initial
`aspire run` repairing dependencies, and Go preview validation should
exercise the same runtime and native-runner paths as other AppHosts.

- Ship a tidy API `go.mod` and generated `go.sum`. Add starter scenarios
checking gitignore, readonly API build, tidy stability, health and
weather responses; remove the Kubernetes manual-tidy workaround.
- Honor the existing `--use-redis-cache false` option for Go.
Omitted/true retains Redis; false omits the integration and resource.
Cover all three scaffold choices with generated-source snapshots and
both HTTP lifecycle variants with Docker E2E.
- Add `go vet` to Go polyglot fixtures, an EF Core fixture, and Foundry
tool/AKS node-pool parity. Exercise configuration-section list getters
in the real Go runtime smoke.
- Add Go to all six native starter-validation platforms, with official
checksum-verified Go installation, timeout budget, and runtime-only
selector edges for Go hosting, Go codegen, and Redis. Linux retains
API/Redis readiness; containerless Windows/macOS use the cache-free
starter with API readiness unchanged.
- Add isolated-local-feed Go/Java scaffolding and restore to the
stabilization smoke without rewriting template pins. Use a genuine
Release bundled driver outside repository mode, cold consumer caches,
and exact hosting-package extraction/provenance checks. Enable Java only
in its subprocess.
- Select languages by name rather than picker position. Restore stale
Java/TypeScript SDK baselines after `aspire add` so the following
`aspire restore` genuinely tests regeneration. Establish the TypeScript
baseline with an actual restore before reading cache metadata.
- Wait for the TypeScript Kubernetes application's actual Ready pod;
exercise the Go Helm namespace/chart-version callback.

Relates to #21202, #21206, #21209, #21210, #21201, and #21108.

### User-facing usage

```bash
# Default retains Redis.
aspire new aspire-go-starter --name GoReady --non-interactive --suppress-agent-init

# Containerless starter: the Go API runs without Redis or a Docker daemon.
aspire new aspire-go-starter --name GoReady --use-redis-cache false --non-interactive --suppress-agent-init
```

Both variants emit `api/go.sum` and an already-tidy API module. Its API
can build with `go build -mod=readonly` without first running the
AppHost's tidy resource.

### Dependencies and remaining acceptance

This PR is independent of #21823. Root-module brownfield E2E #21825
merged into the #21823 feature branch, not directly into main; its
delivery follows #21823. #21806 independently covers other Go
init/build/vet/stable-restore scenarios.

The existing stabilization gate remains disabled
(`STABILIZATION_ENABLED: 'false'`); enabling it needs a separate
CI/policy decision. Current `Aspire.Hosting.Java.csproj` still sets
`SuppressFinalPackageVersion=true` while the Java starter pins the
requested Aspire version. The corrected smoke reaches actual NuGet
restore and exposes that mismatch: `Aspire.Hosting.Java (= 17.0.0)` is
unavailable, the nearest local package is `17.0.0-dev`, and nuget.org
versions are not considered. No pin substitution or Java release-policy
change is included.

Native Linux validation built the genuine stabilized Release CLI/payload
and proved cold Go/Redis package restore and C# restore. This is **not a
clean stabilization pass**: the unsuppressed stable pack first
reproduced NU5104 on existing .NET 11 preview dependencies. The
diagnostic consumer feed downgraded only NU5104 with
`WarningsNotAsErrors=NU5104`; that exception is not committed. The final
smoke correctly exits 1 with only Java in its failure list.

The bounded shared-image fix is separate in #21849; this PR does not
modify image Dockerfiles or the image-build implementation. Its
`b8274fc8e0` image job and all four uploads passed, Infrastructure
passed 2090/2090 including all 13 new image cases, and all 116 CLI
archive E2E jobs passed. Earlier Windows extension fake-timer and DCP
Docker-startup failures were investigated with same-source passing main
controls; the morning live refresh reports its Final Results passing. It
remains independently review-gated. Do not infer actual timeout/fallback
execution from a healthy-mirror image pass.

No action SHA is added or changed. No official Azure DevOps pipeline is
modified. Parent's Go-minimum-version map rule/theory and the other
owner's extension-e2e map block are untouched. The cache option uses
existing embedded CLI inputs; no additional selector rule is required.
The stabilization script remains advisory/disabled rather than being
presented as a regular-PR gate.

This does not close #21108: live Azure Container Apps deployment and
real missing/old-toolchain E2E remain uncovered here. Cross-language
lifecycle redesigns are deferred.

### CI failures diagnosed and fixed

Original-head run
[42116644663](https://github.com/microsoft/aspire/actions/runs/42116644663)
exposed two causes:

- Windows x64/ARM64 and macOS x64/ARM64 starter validation failed
because Redis could not start: the Windows Docker engine pipe was absent
and macOS had no Docker executable. The API correctly waited on its
unhealthy dependency. Those native runners now use the supported
cache-free variant; Linux and Docker E2E retain Redis coverage. No
readiness assertion was removed.
- `TypeScriptCodegenValidationTests` read `.codegen-hash` immediately
after init, which supplies SDK sources but not restore-cache metadata.
An explicit baseline restore now precedes the snapshot and stale-cache
assertions.

**Head `1ab3ad1e30`: all six native starter lanes passed**, including
both Redis-enabled Ubuntu lanes and all four formerly failing
Windows/macOS lanes, in [run
42151619597](https://github.com/microsoft/aspire/actions/runs/42151619597).
Packaged CI subsequently passed
[GoStarterReadinessTests](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127721672515)
(both cache choices),
[GoStarterTemplateTests](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127721672968),
[JavaCodegenValidationTests](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127721672216),
and
[TypeScriptCodegenValidationTests](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127721664876).
All originally failing targeted lanes/classes therefore have passing
coverage.

The shared image job first timed out at 45 minutes twice. Attempt 2
fetched 108 MB from Azure Ubuntu in 41 minutes 29 seconds (43.4 kB/s),
versus a following GitHub download at 80.1 MB/s; GHA cache import also
returned HTTP 400 service-unavailable responses. After #21849's
same-input image control passed with all four uploads, one
evidence-based product revalidation was started. [Attempt 3's image job
passed](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127718680817)
and admitted packaged E2E. No blind retries were made. The successful
infrastructure control used a healthy primary mirror; it does not
demonstrate timeout/fallback execution in that CI run.

**Head `bd30794b9e`: historical failures and final outcome are
distinct.** In [run
42167868738](https://github.com/microsoft/aspire/actions/runs/42167868738),
all six native starter lanes and Go/Java/TypeScript SDK validation
passed. The first [shared image
job](https://github.com/microsoft/aspire/actions/runs/42167868738/job/127729647462)
was canceled after 45 minutes 22 seconds with Azure Ubuntu apt downloads
still running and all four uploads skipped; the initial Linux
tree-actions attempt also failed on a stale immutable Running snapshot.
The morning live refresh subsequently shows [Final Results
passing](https://github.com/microsoft/aspire/actions/runs/42167868738/job/127754717945),
including
[GoStarterReadinessTests](https://github.com/microsoft/aspire/actions/runs/42167868738/job/127747615122).
This session did not initiate retries. A passing later attempt does not
repair the tree-actions race: its separate test-only fix #21854 still
needs independent approval/merge and incorporation before claiming the
race fixed in this checkout.

**Latest head `0bb6569d91` requires fresh checks**, now running in
[42222384609](https://github.com/microsoft/aspire/actions/runs/42222384609).
Both new review findings are corrected: SDK selection accepts only exact
numeric MAJOR.MINOR.PATCH package filenames, and Go starter startup is
inside its existing cleanup scope. Both threads were replied to and
resolved with evidence. Earlier-head CI is not presented as latest-head
proof.

Per the user's explicit authorization, the coordinator enabled normal
native **squash auto-merge** at 05:22:46Z on 2026-11-06. It remains
enabled behind required fresh checks and review requirements. No admin
bypass is authorized or used; the PR is not merged.

### Stabilization review corrections

The smoke now publishes a Release CLI with its real stabilized bundle
outside the checkout, propagates stabilization into child payload
builds, and clears inherited `ASPIRE_REPO_ROOT` for every consumer
invocation. Consumer packages are isolated after the driver build so
build-time extraction cannot hide a missing stable package. Successful
guest restores must produce metadata for the exact hosting pins,
including Redis for Go; provenance is retained in the output.

The real run also exposed two stale smoke assumptions: a pre-owned
parent feed mapping conflicts with scaffolding's project-local channel
policy, and current C# file-based templates use `#:package
Aspire.Hosting.AppHost@<version>`, not the obsolete SDK directive. The
smoke now lets the CLI establish its own authoritative project-scoped
feed mapping and checks the current hosting-package contract. No
customer NuGet policy is changed.

### Validation

- `.\restore.cmd` - passed.
- `dotnet build
tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj
--no-restore --nologo -v quiet` - passed, zero warnings/errors; repeated
successfully after removing temporary fault injection for `0bb6569d91`.
- `dotnet test --project tests\Aspire.Cli.Tests\Aspire.Cli.Tests.csproj
--no-launch-profile -- --filter-class
'*.NewCommandTemplateConfigPersistenceTests' --filter-not-trait
'quarantined=true' --filter-not-trait 'outerloop=true'` - passed, 59
tests including default/true/false Go package and generated-source
assertions. Existing ILCompiler NU1903 advisory was emitted.
- With
`ASPIRE_E2E_CLI_ARCHIVE_DIR=artifacts\readiness-ci\cli-archive-new` and
`ASPIRE_E2E_DOTNET_IMAGE=aspire-e2e-polyglot-base:latest`, `dotnet test
--project
tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj
--no-launch-profile -- --filter-method
'*.RestoreGeneratesSdkFiles_WithConfiguredToolchain' --filter-not-trait
'quarantined=true' --filter-not-trait 'outerloop=true'` - passed, all
four npm/bun/yarn/pnpm cases against exact `1ab3ad1e30` Linux
CLI/packages. The corresponding packaged CI class also passed above.
- With
`ASPIRE_E2E_CLI_ARCHIVE_DIR=artifacts\readiness-ci\cli-archive-new` and
`ASPIRE_E2E_POLYGLOT_GO_IMAGE=aspire-go-e2e-6972:latest`, `dotnet test
--project
tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj
--no-launch-profile -- --filter-method
'*.StarterShipsTidyApiAndServesRequests' --filter-not-trait
'quarantined=true' --filter-not-trait 'outerloop=true'` - 1 passed/1
failed locally against exact `1ab3ad1e30`. Cache-free passed actual
scaffolding, readonly API build, tidy check, API-up, health/weather and
stop without a Docker socket. Redis timed out locally. The unchanged
`2859e199e2` control reproduced the same timeout on this Docker Desktop
host; no workaround or weaker assertion was added. **Both cases
subsequently passed packaged CI**, linked above. After the cleanup-scope
correction, this local command again passed cache-free and failed Redis
on API readiness; the independent CLI stop log confirmed successful
shutdown, while the shared terminal helper observed the prior ERR
prompt. That local run is not claimed green.
- Temporary fault-injection validation used that same E2E command with
only the cache-free case and a shell wrapper returning HTTP 503 for the
Dashboard probe after a real packaged AppHost launch - intentionally
failed. Terminal output showed `dashboard-http-failed`, then cleanup
invocation; the CLI log confirmed `apphost.go stopped successfully` and
exit 0. The wrapper and temporary test edit were removed before the
final clean build/commit. This verifies partial-start cleanup, not a
successful readiness scenario.
- `docker run --rm --entrypoint bash -v "${PWD}:/workspace:ro"
aspire-go-e2e-6972:latest -c "sed 's/\r$//'
/workspace/artifacts/readiness-ci/validate-stable-sdk-selection.sh |
bash"` - passed actual smoke-script preflight cases for missing SDK,
prerelease-only, mixed
stable/dev/preview/rc/build-metadata/four-component filenames,
multi-digit stable versions, and feed paths with spaces. The
instrumented driver received `BundleVersion=17.0.0` from the mixed feed.
Invalid-only feeds exited 1 with the explicit missing-stable diagnostic.
Temporary harness removed after retaining logs; not a new all-language
stabilization pass.
- `dotnet test --project
tests\Infrastructure.Tests\Infrastructure.Tests.csproj
--no-launch-profile -- --filter-class '*.TestTriggerMapTests'
--filter-not-trait 'quarantined=true' --filter-not-trait
'outerloop=true'` - passed, 176 tests.
- `dotnet run --project tools\SelectTests -- --changed-files
artifacts\readiness-changed-files.txt --explain` - passed against the
real graph/map; Go hosting/codegen, Redis, starter checksum and EF Core
fixture inputs select starter-validation/polyglot with zero unattributed
files. Schedule-only deployment remains advisory.
- `$env:DOTNET_ROOT = Join-Path $PWD '.dotnet'; $env:ASPIRE_REPO_ROOT =
$PWD; .\artifacts\bin\Aspire.Cli\Debug\net11.0\aspire.exe restore
--apphost tests\PolyglotAppHosts\<integration>\Go\apphost.go
--non-interactive` - passed separately for
`Aspire.Hosting.EntityFrameworkCore`, `Aspire.Hosting.Foundry`, and
`Aspire.Hosting`, generating actual SDK bindings.
- `docker run --rm --entrypoint bash -v "${PWD}:/workspace"
aspire-go-e2e-6972:latest -c 'set -e; for fixture in
Aspire.Hosting.EntityFrameworkCore Aspire.Hosting.Foundry
Aspire.Hosting; do cd "/workspace/tests/PolyglotAppHosts/$fixture/Go";
gofmt -w apphost.go; go build -buildvcs=false -o /tmp/fixture ./...; go
vet ./...; done'` - passed for all three fixtures.
- `docker run --rm --entrypoint bash -v "${PWD}:/workspace:ro"
aspire-go-e2e-6972:latest -c 'set -e; bash -n
/workspace/.github/workflows/polyglot-validation/test-go.sh
/workspace/.github/workflows/polyglot-validation/test-go-playground.sh
/workspace/eng/scripts/stabilization-smoke-init-restore.sh;
work=$(mktemp -d); trap "rm -rf $work" EXIT; cp
/workspace/src/Aspire.Cli/Templating/Templates/go-starter/api/*
"$work/"; cd "$work"; sed -i "s/{{projectNameLower}}/starter/g" go.mod;
go build -mod=readonly -o /tmp/go-ready-api .; go mod tidy -diff; go vet
./...'` - passed without a dependency-repair command.
- Native Linux `./restore.sh -p:SkipTestProjects=true
-p:SkipPlaygroundProjects=true` - passed with the pinned SDK.
- Native Linux `./dotnet.sh pack
src/Aspire.Hosting.AppHost/Aspire.Hosting.AppHost.csproj -c Release
-p:StabilizePackageVersion=true -p:SkipNativeBuild=true --nologo -v
quiet` - failed as expected on existing NU5104 preview dependencies; not
counted as a stabilization pass.
- To isolate consumer behavior, the diagnostic feed used `for project in
Aspire.Hosting Aspire.Hosting.AppHost Aspire.Hosting.Dotnet
Aspire.TypeSystem Aspire.Hosting.CodeGeneration.Go
Aspire.Hosting.CodeGeneration.Java Aspire.Hosting.Go
Aspire.Hosting.Redis Aspire.Hosting.Java Aspire.Hosting.JavaScript
Aspire.AppHost.Sdk Aspire.ProjectTemplates; do ./dotnet.sh pack
"src/$project/$project.csproj" -c Release
-p:StabilizePackageVersion=true -p:SkipNativeBuild=true
-p:WarningsNotAsErrors=NU5104 --nologo -v quiet; done` - passed. No
suppression is committed; Java remains `17.0.0-dev` by existing policy.
- Native Linux `STABILIZATION_SMOKE_CONFIGURATION=Release
ASPIRE_REPO_ROOT=/work/source bash
eng/scripts/stabilization-smoke-init-restore.sh` - built the genuine
stabilized Release driver/payload and completed the consumer flow.
**Exit 1 is the expected Java policy failure, not an all-language
pass.** Go/Redis `17.0.0` metadata names
`/work/source/artifacts/packages/Release/Shipping`; C# `17.0.0`
restored; Java reached real NuGet restore and rejected its absent stable
pin with no nuget.org fallback. Output assertions confirmed the final
failure list is exactly `java`. Sources were `1ab3ad1e30` plus the final
smoke correction; no mixed PR-stamped CLI was used.
- `bash -n eng/scripts/stabilization-smoke-init-restore.sh`, PowerShell
parser validation of `eng\scripts\cli-starter-validation.ps1`, and `git
diff --check` - passed.
- Full `Infrastructure.Tests.TestTriggerMap` namespace was not passed on
this Windows host: WSL's `bash.exe` cannot find `/bin/bash`; a narrowed
run also encountered readonly `.git` cleanup failures. Not counted as
successful validation.

## Checklist

- Is this feature complete?
  - [ ] Yes. Ready to ship.
  - [x] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
  - [x] Yes
  - [ ] No
- Did you add public API?
  - [ ] Yes
    - If yes, did you have an API Review for it?
      - [ ] Yes
      - [ ] No
- Did you add `<remarks />` and `<code />` elements on your triple slash
comments?
      - [ ] Yes
      - [ ] No
  - [x] No
- Does the change make any security assumptions or guarantees?
  - [ ] Yes
    - If yes, have you done a threat model and had a security review?
      - [ ] Yes
      - [ ] No
  - [x] No

---------

Co-authored-by: IEvangelist <7679720+IEvangelist@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit 5138b2a)
Mitch Denny (mitchdenny) pushed a commit that referenced this pull request Nov 10, 2026
## Description

Go starters should build from their shipped files without an initial
`aspire run` repairing dependencies, and Go preview validation should
exercise the same runtime and native-runner paths as other AppHosts.

- Ship a tidy API `go.mod` and generated `go.sum`. Add starter scenarios
checking gitignore, readonly API build, tidy stability, health and
weather responses; remove the Kubernetes manual-tidy workaround.
- Honor the existing `--use-redis-cache false` option for Go.
Omitted/true retains Redis; false omits the integration and resource.
Cover all three scaffold choices with generated-source snapshots and
both HTTP lifecycle variants with Docker E2E.
- Add `go vet` to Go polyglot fixtures, an EF Core fixture, and Foundry
tool/AKS node-pool parity. Exercise configuration-section list getters
in the real Go runtime smoke.
- Add Go to all six native starter-validation platforms, with official
checksum-verified Go installation, timeout budget, and runtime-only
selector edges for Go hosting, Go codegen, and Redis. Linux retains
API/Redis readiness; containerless Windows/macOS use the cache-free
starter with API readiness unchanged.
- Add isolated-local-feed Go/Java scaffolding and restore to the
stabilization smoke without rewriting template pins. Use a genuine
Release bundled driver outside repository mode, cold consumer caches,
and exact hosting-package extraction/provenance checks. Enable Java only
in its subprocess.
- Select languages by name rather than picker position. Restore stale
Java/TypeScript SDK baselines after `aspire add` so the following
`aspire restore` genuinely tests regeneration. Establish the TypeScript
baseline with an actual restore before reading cache metadata.
- Wait for the TypeScript Kubernetes application's actual Ready pod;
exercise the Go Helm namespace/chart-version callback.

Relates to #21202, #21206, #21209, #21210, #21201, and #21108.

### User-facing usage

```bash
# Default retains Redis.
aspire new aspire-go-starter --name GoReady --non-interactive --suppress-agent-init

# Containerless starter: the Go API runs without Redis or a Docker daemon.
aspire new aspire-go-starter --name GoReady --use-redis-cache false --non-interactive --suppress-agent-init
```

Both variants emit `api/go.sum` and an already-tidy API module. Its API
can build with `go build -mod=readonly` without first running the
AppHost's tidy resource.

### Dependencies and remaining acceptance

This PR is independent of #21823. Root-module brownfield E2E #21825
merged into the #21823 feature branch, not directly into main; its
delivery follows #21823. #21806 independently covers other Go
init/build/vet/stable-restore scenarios.

The existing stabilization gate remains disabled
(`STABILIZATION_ENABLED: 'false'`); enabling it needs a separate
CI/policy decision. Current `Aspire.Hosting.Java.csproj` still sets
`SuppressFinalPackageVersion=true` while the Java starter pins the
requested Aspire version. The corrected smoke reaches actual NuGet
restore and exposes that mismatch: `Aspire.Hosting.Java (= 17.0.0)` is
unavailable, the nearest local package is `17.0.0-dev`, and nuget.org
versions are not considered. No pin substitution or Java release-policy
change is included.

Native Linux validation built the genuine stabilized Release CLI/payload
and proved cold Go/Redis package restore and C# restore. This is **not a
clean stabilization pass**: the unsuppressed stable pack first
reproduced NU5104 on existing .NET 11 preview dependencies. The
diagnostic consumer feed downgraded only NU5104 with
`WarningsNotAsErrors=NU5104`; that exception is not committed. The final
smoke correctly exits 1 with only Java in its failure list.

The bounded shared-image fix is separate in #21849; this PR does not
modify image Dockerfiles or the image-build implementation. Its
`b8274fc8e0` image job and all four uploads passed, Infrastructure
passed 2090/2090 including all 13 new image cases, and all 116 CLI
archive E2E jobs passed. Earlier Windows extension fake-timer and DCP
Docker-startup failures were investigated with same-source passing main
controls; the morning live refresh reports its Final Results passing. It
remains independently review-gated. Do not infer actual timeout/fallback
execution from a healthy-mirror image pass.

No action SHA is added or changed. No official Azure DevOps pipeline is
modified. Parent's Go-minimum-version map rule/theory and the other
owner's extension-e2e map block are untouched. The cache option uses
existing embedded CLI inputs; no additional selector rule is required.
The stabilization script remains advisory/disabled rather than being
presented as a regular-PR gate.

This does not close #21108: live Azure Container Apps deployment and
real missing/old-toolchain E2E remain uncovered here. Cross-language
lifecycle redesigns are deferred.

### CI failures diagnosed and fixed

Original-head run
[42116644663](https://github.com/microsoft/aspire/actions/runs/42116644663)
exposed two causes:

- Windows x64/ARM64 and macOS x64/ARM64 starter validation failed
because Redis could not start: the Windows Docker engine pipe was absent
and macOS had no Docker executable. The API correctly waited on its
unhealthy dependency. Those native runners now use the supported
cache-free variant; Linux and Docker E2E retain Redis coverage. No
readiness assertion was removed.
- `TypeScriptCodegenValidationTests` read `.codegen-hash` immediately
after init, which supplies SDK sources but not restore-cache metadata.
An explicit baseline restore now precedes the snapshot and stale-cache
assertions.

**Head `1ab3ad1e30`: all six native starter lanes passed**, including
both Redis-enabled Ubuntu lanes and all four formerly failing
Windows/macOS lanes, in [run
42151619597](https://github.com/microsoft/aspire/actions/runs/42151619597).
Packaged CI subsequently passed
[GoStarterReadinessTests](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127721672515)
(both cache choices),
[GoStarterTemplateTests](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127721672968),
[JavaCodegenValidationTests](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127721672216),
and
[TypeScriptCodegenValidationTests](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127721664876).
All originally failing targeted lanes/classes therefore have passing
coverage.

The shared image job first timed out at 45 minutes twice. Attempt 2
fetched 108 MB from Azure Ubuntu in 41 minutes 29 seconds (43.4 kB/s),
versus a following GitHub download at 80.1 MB/s; GHA cache import also
returned HTTP 400 service-unavailable responses. After #21849's
same-input image control passed with all four uploads, one
evidence-based product revalidation was started. [Attempt 3's image job
passed](https://github.com/microsoft/aspire/actions/runs/42151619597/job/127718680817)
and admitted packaged E2E. No blind retries were made. The successful
infrastructure control used a healthy primary mirror; it does not
demonstrate timeout/fallback execution in that CI run.

**Head `bd30794b9e`: historical failures and final outcome are
distinct.** In [run
42167868738](https://github.com/microsoft/aspire/actions/runs/42167868738),
all six native starter lanes and Go/Java/TypeScript SDK validation
passed. The first [shared image
job](https://github.com/microsoft/aspire/actions/runs/42167868738/job/127729647462)
was canceled after 45 minutes 22 seconds with Azure Ubuntu apt downloads
still running and all four uploads skipped; the initial Linux
tree-actions attempt also failed on a stale immutable Running snapshot.
The morning live refresh subsequently shows [Final Results
passing](https://github.com/microsoft/aspire/actions/runs/42167868738/job/127754717945),
including
[GoStarterReadinessTests](https://github.com/microsoft/aspire/actions/runs/42167868738/job/127747615122).
This session did not initiate retries. A passing later attempt does not
repair the tree-actions race: its separate test-only fix #21854 still
needs independent approval/merge and incorporation before claiming the
race fixed in this checkout.

**Latest head `0bb6569d91` requires fresh checks**, now running in
[42222384609](https://github.com/microsoft/aspire/actions/runs/42222384609).
Both new review findings are corrected: SDK selection accepts only exact
numeric MAJOR.MINOR.PATCH package filenames, and Go starter startup is
inside its existing cleanup scope. Both threads were replied to and
resolved with evidence. Earlier-head CI is not presented as latest-head
proof.

Per the user's explicit authorization, the coordinator enabled normal
native **squash auto-merge** at 05:22:46Z on 2026-11-06. It remains
enabled behind required fresh checks and review requirements. No admin
bypass is authorized or used; the PR is not merged.

### Stabilization review corrections

The smoke now publishes a Release CLI with its real stabilized bundle
outside the checkout, propagates stabilization into child payload
builds, and clears inherited `ASPIRE_REPO_ROOT` for every consumer
invocation. Consumer packages are isolated after the driver build so
build-time extraction cannot hide a missing stable package. Successful
guest restores must produce metadata for the exact hosting pins,
including Redis for Go; provenance is retained in the output.

The real run also exposed two stale smoke assumptions: a pre-owned
parent feed mapping conflicts with scaffolding's project-local channel
policy, and current C# file-based templates use `#:package
Aspire.Hosting.AppHost@<version>`, not the obsolete SDK directive. The
smoke now lets the CLI establish its own authoritative project-scoped
feed mapping and checks the current hosting-package contract. No
customer NuGet policy is changed.

### Validation

- `.\restore.cmd` - passed.
- `dotnet build
tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj
--no-restore --nologo -v quiet` - passed, zero warnings/errors; repeated
successfully after removing temporary fault injection for `0bb6569d91`.
- `dotnet test --project tests\Aspire.Cli.Tests\Aspire.Cli.Tests.csproj
--no-launch-profile -- --filter-class
'*.NewCommandTemplateConfigPersistenceTests' --filter-not-trait
'quarantined=true' --filter-not-trait 'outerloop=true'` - passed, 59
tests including default/true/false Go package and generated-source
assertions. Existing ILCompiler NU1903 advisory was emitted.
- With
`ASPIRE_E2E_CLI_ARCHIVE_DIR=artifacts\readiness-ci\cli-archive-new` and
`ASPIRE_E2E_DOTNET_IMAGE=aspire-e2e-polyglot-base:latest`, `dotnet test
--project
tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj
--no-launch-profile -- --filter-method
'*.RestoreGeneratesSdkFiles_WithConfiguredToolchain' --filter-not-trait
'quarantined=true' --filter-not-trait 'outerloop=true'` - passed, all
four npm/bun/yarn/pnpm cases against exact `1ab3ad1e30` Linux
CLI/packages. The corresponding packaged CI class also passed above.
- With
`ASPIRE_E2E_CLI_ARCHIVE_DIR=artifacts\readiness-ci\cli-archive-new` and
`ASPIRE_E2E_POLYGLOT_GO_IMAGE=aspire-go-e2e-6972:latest`, `dotnet test
--project
tests\Aspire.Cli.EndToEnd.Tests\Aspire.Cli.EndToEnd.Tests.csproj
--no-launch-profile -- --filter-method
'*.StarterShipsTidyApiAndServesRequests' --filter-not-trait
'quarantined=true' --filter-not-trait 'outerloop=true'` - 1 passed/1
failed locally against exact `1ab3ad1e30`. Cache-free passed actual
scaffolding, readonly API build, tidy check, API-up, health/weather and
stop without a Docker socket. Redis timed out locally. The unchanged
`2859e199e2` control reproduced the same timeout on this Docker Desktop
host; no workaround or weaker assertion was added. **Both cases
subsequently passed packaged CI**, linked above. After the cleanup-scope
correction, this local command again passed cache-free and failed Redis
on API readiness; the independent CLI stop log confirmed successful
shutdown, while the shared terminal helper observed the prior ERR
prompt. That local run is not claimed green.
- Temporary fault-injection validation used that same E2E command with
only the cache-free case and a shell wrapper returning HTTP 503 for the
Dashboard probe after a real packaged AppHost launch - intentionally
failed. Terminal output showed `dashboard-http-failed`, then cleanup
invocation; the CLI log confirmed `apphost.go stopped successfully` and
exit 0. The wrapper and temporary test edit were removed before the
final clean build/commit. This verifies partial-start cleanup, not a
successful readiness scenario.
- `docker run --rm --entrypoint bash -v "${PWD}:/workspace:ro"
aspire-go-e2e-6972:latest -c "sed 's/\r$//'
/workspace/artifacts/readiness-ci/validate-stable-sdk-selection.sh |
bash"` - passed actual smoke-script preflight cases for missing SDK,
prerelease-only, mixed
stable/dev/preview/rc/build-metadata/four-component filenames,
multi-digit stable versions, and feed paths with spaces. The
instrumented driver received `BundleVersion=17.0.0` from the mixed feed.
Invalid-only feeds exited 1 with the explicit missing-stable diagnostic.
Temporary harness removed after retaining logs; not a new all-language
stabilization pass.
- `dotnet test --project
tests\Infrastructure.Tests\Infrastructure.Tests.csproj
--no-launch-profile -- --filter-class '*.TestTriggerMapTests'
--filter-not-trait 'quarantined=true' --filter-not-trait
'outerloop=true'` - passed, 176 tests.
- `dotnet run --project tools\SelectTests -- --changed-files
artifacts\readiness-changed-files.txt --explain` - passed against the
real graph/map; Go hosting/codegen, Redis, starter checksum and EF Core
fixture inputs select starter-validation/polyglot with zero unattributed
files. Schedule-only deployment remains advisory.
- `$env:DOTNET_ROOT = Join-Path $PWD '.dotnet'; $env:ASPIRE_REPO_ROOT =
$PWD; .\artifacts\bin\Aspire.Cli\Debug\net11.0\aspire.exe restore
--apphost tests\PolyglotAppHosts\<integration>\Go\apphost.go
--non-interactive` - passed separately for
`Aspire.Hosting.EntityFrameworkCore`, `Aspire.Hosting.Foundry`, and
`Aspire.Hosting`, generating actual SDK bindings.
- `docker run --rm --entrypoint bash -v "${PWD}:/workspace"
aspire-go-e2e-6972:latest -c 'set -e; for fixture in
Aspire.Hosting.EntityFrameworkCore Aspire.Hosting.Foundry
Aspire.Hosting; do cd "/workspace/tests/PolyglotAppHosts/$fixture/Go";
gofmt -w apphost.go; go build -buildvcs=false -o /tmp/fixture ./...; go
vet ./...; done'` - passed for all three fixtures.
- `docker run --rm --entrypoint bash -v "${PWD}:/workspace:ro"
aspire-go-e2e-6972:latest -c 'set -e; bash -n
/workspace/.github/workflows/polyglot-validation/test-go.sh
/workspace/.github/workflows/polyglot-validation/test-go-playground.sh
/workspace/eng/scripts/stabilization-smoke-init-restore.sh;
work=$(mktemp -d); trap "rm -rf $work" EXIT; cp
/workspace/src/Aspire.Cli/Templating/Templates/go-starter/api/*
"$work/"; cd "$work"; sed -i "s/{{projectNameLower}}/starter/g" go.mod;
go build -mod=readonly -o /tmp/go-ready-api .; go mod tidy -diff; go vet
./...'` - passed without a dependency-repair command.
- Native Linux `./restore.sh -p:SkipTestProjects=true
-p:SkipPlaygroundProjects=true` - passed with the pinned SDK.
- Native Linux `./dotnet.sh pack
src/Aspire.Hosting.AppHost/Aspire.Hosting.AppHost.csproj -c Release
-p:StabilizePackageVersion=true -p:SkipNativeBuild=true --nologo -v
quiet` - failed as expected on existing NU5104 preview dependencies; not
counted as a stabilization pass.
- To isolate consumer behavior, the diagnostic feed used `for project in
Aspire.Hosting Aspire.Hosting.AppHost Aspire.Hosting.Dotnet
Aspire.TypeSystem Aspire.Hosting.CodeGeneration.Go
Aspire.Hosting.CodeGeneration.Java Aspire.Hosting.Go
Aspire.Hosting.Redis Aspire.Hosting.Java Aspire.Hosting.JavaScript
Aspire.AppHost.Sdk Aspire.ProjectTemplates; do ./dotnet.sh pack
"src/$project/$project.csproj" -c Release
-p:StabilizePackageVersion=true -p:SkipNativeBuild=true
-p:WarningsNotAsErrors=NU5104 --nologo -v quiet; done` - passed. No
suppression is committed; Java remains `17.0.0-dev` by existing policy.
- Native Linux `STABILIZATION_SMOKE_CONFIGURATION=Release
ASPIRE_REPO_ROOT=/work/source bash
eng/scripts/stabilization-smoke-init-restore.sh` - built the genuine
stabilized Release driver/payload and completed the consumer flow.
**Exit 1 is the expected Java policy failure, not an all-language
pass.** Go/Redis `17.0.0` metadata names
`/work/source/artifacts/packages/Release/Shipping`; C# `17.0.0`
restored; Java reached real NuGet restore and rejected its absent stable
pin with no nuget.org fallback. Output assertions confirmed the final
failure list is exactly `java`. Sources were `1ab3ad1e30` plus the final
smoke correction; no mixed PR-stamped CLI was used.
- `bash -n eng/scripts/stabilization-smoke-init-restore.sh`, PowerShell
parser validation of `eng\scripts\cli-starter-validation.ps1`, and `git
diff --check` - passed.
- Full `Infrastructure.Tests.TestTriggerMap` namespace was not passed on
this Windows host: WSL's `bash.exe` cannot find `/bin/bash`; a narrowed
run also encountered readonly `.git` cleanup failures. Not counted as
successful validation.

## Checklist

- Is this feature complete?
  - [ ] Yes. Ready to ship.
  - [x] No. Follow-up changes expected.
- Are you including unit tests for the changes and scenario tests if
relevant?
  - [x] Yes
  - [ ] No
- Did you add public API?
  - [ ] Yes
    - If yes, did you have an API Review for it?
      - [ ] Yes
      - [ ] No
- Did you add `<remarks />` and `<code />` elements on your triple slash
comments?
      - [ ] Yes
      - [ ] No
  - [x] No
- Does the change make any security assumptions or guarantees?
  - [ ] Yes
    - If yes, have you done a threat model and had a security review?
      - [ ] Yes
      - [ ] No
  - [x] No

---------

Co-authored-by: IEvangelist <7679720+IEvangelist@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
(cherry picked from commit 5138b2a)
clearTimeout(grace);
const clientExitCode = code ?? (signal ? 128 + constants.signals[signal] : 1);
// An external kill/OOM during cleanup is not acknowledged cancellation.
process.exit(forcedKill || clientExitCode === 137 ? 137 : cancellationExitCode || (deadlineReached ? 124 : clientExitCode));

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[automated] [P2] Fail closed when the Buildx plugin is killed during cleanup

The supervised child is Docker, not Buildx itself. With Docker 28.0.4 (the version in this PR's CI), a SIGKILLed plugin makes Docker exit 255, not 137. Once the deadline has fired, this expression converts that to 124, so build_with_mirror_retry starts fallback and can return success without confirmed graceful cancellation.

I reproduced this through the production helper and supervisor using real Docker plus a fixture Buildx plugin that kills itself on deadline TERM: PLUGIN_KILLED_DURING_CLEANUP -> FALLBACK_RAN -> helper exit 0. This demonstrates the exit-classification bug, not actual surviving BuildKit work.

Handle the Docker/plugin termination path before converting cleanup exits to timeout, and cover that process boundary; the current direct-client SIGKILL regression only exercises exit 137.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants