Skip to content

Don't probe the container runtime at startup when the app has no containers - #21957

Merged
David Fowler (davidfowl) merged 5 commits into
mainfrom
aspireagent/dcp-version-check-without-containers
Nov 10, 2026
Merged

David Fowler (davidfowl) merged 5 commits into
mainfrom
aspireagent/dcp-version-check-without-containers

Conversation

@davidfowl

@davidfowl David Fowler (davidfowl) commented Nov 9, 2026 •

Copy link
Copy Markdown
Collaborator

Description

What failed. On Windows runners, the Deno functional tests (DenoTelemetryFunctionalTests.VerifyDenoAppExportsNativeTelemetryToDashboard, DenoFunctionalTests.VerifyDenoAppDirectExecutionWorks, DenoFunctionalTests.VerifyDenoAppTaskScriptWorks) and other DCP-backed fixtures fail in InitializeAsync with:

Aspire.Hosting.DistributedApplicationException : Application orchestrator dependency check returned an error: The operation has timed out.
dcp.info.os-executor Context expired, stopping process... {"Command": "C:\\Windows\\system32\\docker.exe", "Args": ["--version"]}

Example runs: 42157416383 (Hosting.JavaScript), 42076556098 (Hosting.JavaScript), 41910295423 (Hosting-2, Hosting.DotnetTool), 42088737959 (Hosting.Testing), 42253714471 (Hosting-4).

Why. DcpHost.EnsureDcpContainerRuntimeAsync and DcpExecutor.RunApplicationAsync ran dcp info on every run-mode start. dcp info probes the container runtime CLI (docker --version). When that CLI is slow or hung on a busy Windows runner, DCP spends about 10s on the probe and then about 15s stopping it. That lands right on Aspire's 25s DependencyCheckTimeout, so startup fails even for apps with no containers. The Deno fixtures only have executables, so the container-runtime result is never used.

How it's fixed.

  • IDcpDependencyCheckService gains GetDcpVersionAsync. It validates the DCP install and minimum version with dcp version, which prints the same {"version": ...} JSON but doesn't probe the container runtime. dcp version is supported back to the minimum DCP 0.23.3. DcpDependencyCheck keeps one nullable _dcpInfo with the last successful result: a version check reuses any cached result, and a full-info check reuses it only when it includes container status. A failed or cancelled probe leaves the cache unchanged, so it is retried.
  • Empty stdout or a JSON null from DCP now throws a dependency-check exception instead of returning null, so startup can't continue without a validated DCP version.
  • DcpHost and DcpExecutor use GetDcpVersionAsync when the model has no container resources. Apps with containers keep the full dcp info check and still fail loudly when the runtime is unhealthy.
  • HostUrl and ContainerNetworkEndpointProvisioner still call dcp info, but only for container networks, so executables never reach them.

This is a product-layer fix, so it covers every executable-only app and test fixture, not just Deno. It complements the test-only approaches in #21928 and #21939, which avoid DCP in tests that don't need it. Neither of those covers the Deno tests.

Fixes #21771
Fixes #21848
Related: #20850 (same Deno fixture; that report is a different DCP SQLite startup crash and is not addressed here)

Confidence

High that this removes the docker --version timeout for executable-only apps. On that path the failing call is no longer made, as the paired unit tests and the real-DCP repro with a hung docker show. The main limit is volume: the probe signature has only 4 CI hits in 14 days, and none since 11-06 21:06Z, so CI can't yet confirm the fix by itself. Most of the Deno failures in the 14-day window have other causes (see below). Those stopped after the DCP 0.27.3 bump and are not addressed here.

How we stop this recurring: startup for apps without containers no longer depends on how responsive the container-runtime CLI is. The new DcpHost and DcpExecutor tests fail if a no-container path starts calling dcp info again.

Failure stats (14d, attempt 1)

Window: 10-26 10:00Z to 11-09 10:00Z, ci.yml, attempt 1 only. The denominator is the 757 runs where a Hosting.JavaScript job executed and finished (157 main + 600 PR). One PR run is excluded as PR-caused: every Node and Deno test failed on both OSes, and the reruns kept failing.

test failed runs (main + PR) rate passed on rerun
DenoTelemetryFunctionalTests.VerifyDenoAppExportsNativeTelemetryToDashboard 25 (6 + 19) 3.3% 14
DenoFunctionalTests.VerifyDenoAppDirectExecutionWorks 25 (7 + 18) 3.3% 11
DenoFunctionalTests.VerifyDenoAppTaskScriptWorks 25 (7 + 18) 3.3% 11

46 runs had at least one Deno failure (11 main, 35 PR). By cause:

  • DCP probe timeout (dependency check returned an error / docker.exe --version), which this PR fixes: 4 runs (2 main, 2 PR), 11-05 17:38Z to 11-06 21:06Z, all three tests each time. Both PR runs passed on rerun; the main runs weren't rerun.
  • 2-minute Polly timeout in fixture init: 26 runs, last 11-03 00:36Z.
  • Service … should have valid address at this point: 16 runs, last 11-02 15:08Z.

The last two causes stopped after the DCP 0.27.3 bump (#21706, merged 11-03 05:38Z). Since 11-03, every Deno failure has been the probe signature (4/334 runs, 1.2%). The tracker #21600 counts 20/17/17 over all 1,173 decided ci.yml runs. Its numerator and denominator differ from this job-level count. The same probe signature also hits other executable-only fixtures (Hosting.Testing, Hosting.DotnetTool, Hosting-N); see #21771. Those are not counted here.

Runs: 42076556098 (main, probe), 42157416383 (main, probe), 42246915065 (PR, probe), 42293916898 (PR, probe), 41554110222 (main, older Polly-timeout cause).

  • Still hits main? Last main hit was 11-06 03:34Z (probe). There have been no hits in the 68 runs since 11-06 21:06Z (13 of them on main).
  • Local before/after: the no-container regression tests fail without the fix and pass 3/3 with it. With a hung docker on PATH, all 3 real Deno tests fail fixture init without the fix. With the fix, VerifyDenoAppDirectExecutionWorks and VerifyDenoAppTaskScriptWorks pass 3/3, and the telemetry test gets past fixture init (details under Evidence).
  • This PR's CI: on the current head (rebased onto main), Final Results passed on attempt 1 (42749913164), with no reruns. Every job passed, including Hosting.JavaScript (where the Deno tests run) and Hosting-1..6 on Windows and Ubuntu. The only red run on the head is the Copilot code-review bot, which timed out waiting for a runner; it is not a CI test job.

Verdict: MODERATE. The fix removes the exact call behind all 4 probe-signature failures, and a deterministic local repro flips from fail to pass. But with only 4 hits in 14 days and none since 11-06, CI can't confirm it yet.

Evidence (paired, local Linux)

Root cause with real DCP 0.27.3 and a fake docker that hangs:

command result
dcp info --container-runtime docker 25s, Context expired, stopping process... docker --version (the CI signature)
dcp version 0s, {"version":"0.27.3",...}

Re-checked with DCP 0.27.4 (current main): dcp version returns only version fields, and dcp info always includes a containers block, even when docker fails or isn't on PATH. So Containers != null reliably identifies a cached dcp info result.

Regression tests (Aspire.Hosting.Tests):

test without fix with fix
DcpHostNotificationTests.EnsureDcpContainerRuntimeAsync_WithoutContainers_DoesNotWaitForContainerRuntimeProbe ❌ dependency check returned an error: The operation has timed out ✅ (3/3)
DcpExecutorTests.RunApplicationAsync_WithoutContainers_DoesNotProbeContainerRuntime ❌ GetDcpInfoAsync was called ✅ (3/3)
DcpHostNotificationTests.EnsureDcpContainerRuntimeAsync_WithContainers_ReportsContainerRuntimeProbeTimeout ✅ ✅ (containers still fail loudly)
DcpExecutorTests.RunApplicationAsync_WithContainers_ProbesContainerRuntime ✅ ✅
DcpHostNotificationTests.GetDcpVersionAsync_RetriesAfterFailedCheck ❌ (failed probe cached as null) ✅
DcpHostNotificationTests.DependencyCheck_ThrowsWhenDcpReturnsNoInformation (empty / null output, version and info) ❌ (returned null) ✅

All 377 tests in DcpHostNotificationTests and DcpExecutorTests pass on the current head.

The real Deno tests, with a hung docker on PATH and ASPIRE_DEPENDENCY_CHECK_TIMEOUT=8 to make the race deterministic:

  • Without fix: all 3 Deno tests fail in fixture init with Application orchestrator dependency check returned an error: The operation has timed out.
  • With fix: both fixtures initialize, and VerifyDenoAppDirectExecutionWorks and VerifyDenoAppTaskScriptWorks pass 3/3. The telemetry test gets past fixture init. Its dashboard-query step times out in this sandbox on main as well, with or without the hung docker, so that part is environmental and unrelated.

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
    • No

🤖 Posted by AspireAgent (AI Agent)

DcpHost and DcpExecutor ran 'dcp info' on every run-mode start. 'dcp info'
probes the container runtime CLI (docker --version), which can hang on busy
Windows runners past the 25s dependency check timeout and fail startup even
for apps without containers (#21771, #21848).

Add IDcpDependencyCheckService.GetDcpVersionAsync, which validates the DCP
installation and minimum version with 'dcp version' (no container probe), and
use it when the model has no container resources. Apps with containers keep
the full 'dcp info' check.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
RunApplicationAsync must use 'dcp version' when the model has no
containers, otherwise it runs 'dcp info' right after DcpHost and can still
hang on an unresponsive container runtime CLI (#21771).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@davidfowl David Fowler (davidfowl) added area-orchestrator agent:working An AI agent is actively working on this issue or PR labels Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21957

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21957"

@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

DCP checks mark themselves complete before succeeding, allowing failed probes to make subsequent calls return null without validation, retrying, or reusing a valid cached version result.

2 open issues
What changed in this PR

This PR removes container-runtime probing from executable-only app startup while preserving DCP installation and version validation. Apps with containers retain the full runtime health check.

Changes:

  • Adds a cached dcp version check that avoids probing Docker or Podman.
  • Routes host and executor startup checks according to whether the model contains containers.
  • Adds regression tests and dependency-check call counters.
File Description
tests/​Aspire.Hosting.TestUtilities/​Dcp/​TestDcpDependencyCheckService.cs Adds version-check support and thread-safe call counters.
tests/​Aspire.Hosting.Tests/​Dcp/​DcpHostNotificationTests.cs Tests startup behavior with a hanging container-runtime probe.
tests/​Aspire.Hosting.Tests/​Dcp/​DcpExecutorTests.cs Verifies probe selection for models with and without containers.
src/​Aspire.Hosting/​Dcp/​IDcpDependencyCheckService.cs Adds the version-only dependency-check contract.
src/​Aspire.Hosting/​Dcp/​DcpHost.cs Skips runtime probing for models without containers.
src/​Aspire.Hosting/​Dcp/​DcpExecutor.cs Selects version-only or full checks during execution.
src/​Aspire.Hosting/​Dcp/​DcpDependencyCheck.cs Implements version-check caching and shares command execution logic.

Review effort: Balanced

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The Windows tests launch a .cmd file without a command interpreter, causing a startup failure that breaks the no-container case and can falsely satisfy the container case’s generic error assertion instead of verifying a timeout.

2 open issues
1 previously missed issue

In code that hasn't changed since last review

Medium severity Windows tests launch the .cmd fake without a command interpreter

tests/​Aspire.Hosting.Tests/​Dcp/​DcpHostNotificationTests.cs:833

On Windows, ProcessUtil.Run launches this .cmd file with UseShellExecute = false and no command interpreter, so the no-container test fails at process startup instead of checking dcp version. The container test can pass for the same wrong reason because it only asserts the generic dependency-error prefix. Use an executable fake or a test launch seam that invokes cmd.exe /c, and assert that the container failure is actually a timeout.

Review effort: Balanced

A failed or cancelled 'dcp version' probe no longer leaves the check marked
done, and a failed 'dcp info' check no longer short-circuits the version-only
path.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the change to skip container-runtime probing in applications without containers.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

@davidfowl David Fowler (davidfowl) added the ci-health Fixes CI flakiness or CI infrastructure (AspireAgent queue) label Nov 9, 2026 — with Copilot SWE Agent
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Copy link
Copy Markdown
Collaborator Author

Devin Breshears (@dbreshears), could you review this as the orchestrator area owner? It's ready, and Final Results is green on the current head. It skips the dcp info container-runtime probe at startup when the app has no containers. That probe's docker --version timeout has been behind every Deno functional test failure since 11-03. The PR body has the 14-day stats.

GitHub wouldn't let me add you or the area-orchestrator team as formal reviewers (neither is a collaborator on this repo), so I'm asking here instead.

🤖 Posted by AspireAgent (AI Agent)

@davidfowl David Fowler (davidfowl) removed the agent:working An AI agent is actively working on this issue or PR label Nov 9, 2026

@danegsta David Negstad (danegsta) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two comments: one cache simplification and one dependency-check error-handling concern.

@davidfowl David Fowler (davidfowl) added the agent:working An AI agent is actively working on this issue or PR label Nov 10, 2026 — with Copilot SWE Agent

David Fowler (davidfowl) commented Nov 10, 2026 •

Copy link
Copy Markdown
Collaborator Author

AspireAgent stopped work here: review threads are answered, the description is updated, CI is green on the head, and the PR is waiting on human review.

🤖 Posted by AspireAgent (AI Agent)

Collapse the separate version and info caches into one nullable _dcpInfo
that holds the last successful check. GetDcpVersionAsync reuses any cached
result; GetDcpInfoAsync reuses it only when it includes container status.

Empty stdout or a JSON null from DCP now throws a dependency-check
exception instead of returning null, so startup can't continue without a
validated DCP version.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@davidfowl
David Fowler (davidfowl) force-pushed the aspireagent/dcp-version-check-without-containers branch from e6b2517 to 26adb1f Compare November 10, 2026 03:30
@aspire-repo-bot
aspire-repo-bot Bot requested a balanced review from Copilot November 10, 2026 03:33
@github-actions

Copy link
Copy Markdown
Contributor

Tests selector

52 / 101 PR test projects · 4 PR jobs, from 7 changed files.

Selected PR test projects (52 / 101)

Aspire.Cli.EndToEnd.Tests, Aspire.Hosting.Analyzers.Tests, Aspire.Hosting.Azure.EntraId.Tests, Aspire.Hosting.Azure.Kubernetes.Tests, Aspire.Hosting.Azure.Kusto.Tests, Aspire.Hosting.Azure.Provisioning.Tests, Aspire.Hosting.Azure.Tests, Aspire.Hosting.Blazor.Tests, Aspire.Hosting.Browsers.Tests, Aspire.Hosting.CodeGeneration.Go.Tests, Aspire.Hosting.CodeGeneration.Java.Tests, Aspire.Hosting.CodeGeneration.Python.Tests, Aspire.Hosting.CodeGeneration.Rust.Tests, Aspire.Hosting.CodeGeneration.TypeScript.Tests, Aspire.Hosting.Containers.Tests, Aspire.Hosting.DevTunnels.Tests, Aspire.Hosting.Docker.Tests, Aspire.Hosting.Dotnet.Tests, Aspire.Hosting.DotnetTool.Tests, Aspire.Hosting.EntityFrameworkCore.Tests, Aspire.Hosting.Foundry.Tests, Aspire.Hosting.Garnet.Tests, Aspire.Hosting.Go.Tests, Aspire.Hosting.Java.Tests, Aspire.Hosting.JavaScript.Tests, Aspire.Hosting.Kafka.Tests, Aspire.Hosting.Keycloak.Tests, Aspire.Hosting.Kubernetes.Tests, Aspire.Hosting.Maui.Tests, Aspire.Hosting.Milvus.Tests, Aspire.Hosting.MongoDB.Tests, Aspire.Hosting.MySql.Tests, Aspire.Hosting.Nats.Tests, Aspire.Hosting.OpenAI.Tests, Aspire.Hosting.Oracle.Tests, Aspire.Hosting.Orleans.Tests, Aspire.Hosting.PostgreSQL.Tests, Aspire.Hosting.Python.Tests, Aspire.Hosting.Qdrant.Tests, Aspire.Hosting.RabbitMQ.Tests, Aspire.Hosting.Radius.Tests, Aspire.Hosting.Redis.Tests, Aspire.Hosting.RemoteHost.Tests, Aspire.Hosting.Rust.Tests, Aspire.Hosting.Seq.Tests, Aspire.Hosting.SqlServer.Tests, Aspire.Hosting.Testing.Tests, Aspire.Hosting.Tests, Aspire.Hosting.Valkey.Tests, Aspire.Hosting.Yarp.Tests, Aspire.Playground.Tests, Aspire.Templates.Tests

Selected PR jobs (4)

cli-starter-validation, extension-e2e, polyglot, typescript-api-compat


How these were chosen — grouped by what changed

⚠️ 34 of the 52 selected test projects come from a single change — tests/Aspire.Hosting.TestUtilities/Dcp/TestDcpDependencyCheckService.cs.

🧪 tests/Aspire.Hosting.TestUtilities/Dcp/TestDcpDependencyCheckService.cs (changed test)
→ 34 via the project graph

show 34

Aspire.Hosting.Azure.EntraId.Tests, Aspire.Hosting.Azure.Kubernetes.Tests, Aspire.Hosting.Azure.Kusto.Tests, Aspire.Hosting.Browsers.Tests, Aspire.Hosting.Containers.Tests, Aspire.Hosting.DevTunnels.Tests, Aspire.Hosting.Docker.Tests, Aspire.Hosting.DotnetTool.Tests, Aspire.Hosting.EntityFrameworkCore.Tests, Aspire.Hosting.Foundry.Tests, Aspire.Hosting.Garnet.Tests, Aspire.Hosting.Go.Tests, Aspire.Hosting.Java.Tests, Aspire.Hosting.JavaScript.Tests, Aspire.Hosting.Kafka.Tests, Aspire.Hosting.Keycloak.Tests, Aspire.Hosting.Kubernetes.Tests, Aspire.Hosting.Milvus.Tests, Aspire.Hosting.MongoDB.Tests, Aspire.Hosting.MySql.Tests, Aspire.Hosting.Nats.Tests, Aspire.Hosting.OpenAI.Tests, Aspire.Hosting.Oracle.Tests, Aspire.Hosting.Orleans.Tests, Aspire.Hosting.PostgreSQL.Tests, Aspire.Hosting.Python.Tests, Aspire.Hosting.Qdrant.Tests, Aspire.Hosting.RabbitMQ.Tests, Aspire.Hosting.Redis.Tests, Aspire.Hosting.Rust.Tests, Aspire.Hosting.Seq.Tests, Aspire.Hosting.SqlServer.Tests, Aspire.Hosting.Valkey.Tests, Aspire.Hosting.Yarp.Tests

🔧 src/Aspire.Hosting/Dcp/DcpDependencyCheck.cs (changed source)
→ 12 via the project graph: Aspire.Hosting.Analyzers.Tests (2 hops), Aspire.Hosting.Azure.Provisioning.Tests (3 hops), Aspire.Hosting.Azure.Tests, Aspire.Hosting.CodeGeneration.Go.Tests, Aspire.Hosting.CodeGeneration.Java.Tests, Aspire.Hosting.CodeGeneration.Python.Tests, Aspire.Hosting.CodeGeneration.Rust.Tests, Aspire.Hosting.CodeGeneration.TypeScript.Tests, Aspire.Hosting.Maui.Tests, Aspire.Hosting.RemoteHost.Tests, Aspire.Hosting.Testing.Tests (2 hops), Aspire.Playground.Tests

🧪 tests/Aspire.Hosting.Tests/Dcp/DcpExecutorTests.cs (changed test)
→ 1 directly: Aspire.Hosting.Tests
→ 3 via the project graph: Aspire.Hosting.Blazor.Tests, Aspire.Hosting.Dotnet.Tests, Aspire.Hosting.Radius.Tests

📦 affected project Aspire.Hosting.Dotnet
→ 2 tests: Aspire.Cli.EndToEnd.Tests, Aspire.Templates.Tests

📦 affected project Aspire.Hosting
→ 1 test: Aspire.Cli.EndToEnd.Tests

🧪 tests/Aspire.Hosting.Tests/Dcp/DcpHostNotificationTests.cs (changed test)
→ 1 directly: Aspire.Hosting.Tests

Job reasons

Job Triggered by
cli-starter-validation affected project Aspire.Hosting.Dotnet
extension-e2e • src/Aspire.Hosting/Dcp/DcpDependencyCheck.cs, src/Aspire.Hosting/Dcp/DcpExecutor.cs, src/Aspire.Hosting/Dcp/DcpHost.cs, src/Aspire.Hosting/Dcp/IDcpDependencyCheckService.cs
• affected project Aspire.Hosting
polyglot • affected project Aspire.Hosting
• affected project Aspire.Hosting.Azure.Provisioning.KeyVault
typescript-api-compat affected project Aspire.Hosting

Selection computed for commit 26adb1f.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

4 open issues

Review effort: Lite

Comment thread src/Aspire.Hosting/Dcp/DcpHost.cs Resolved
@davidfowl
David Fowler (davidfowl) merged commit e6a9cad into main Nov 10, 2026
358 of 359 checks passed
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

✅ No documentation update needed.

Branch: proven false positive (closest category: internal_refactor).

  • Triggered signals (1): pr_body_has_cli_flag_mention. Evidence: <pr-body> matched "Args": ["--version"] in a quoted CI error log (docker.exe --version), which is not an Aspire CLI flag.
  • The PR changes internal DCP startup behavior. Executable-only apps now run dcp version instead of dcp info, which avoids a container runtime probe timeout. It adds IDcpDependencyCheckService.GetDcpVersionAsync (internal hosting plumbing) and introduces no new user-facing option, config key, API, or documented behavior.
  • Apps with containers keep the full check and the same failure behavior.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-orchestrator ci-health Fixes CI flakiness or CI infrastructure (AspireAgent queue) flaky-test

Projects

None yet

4 participants