Repository navigation
Keep Java container env var tests from running the real dcp info - #21939
Conversation
AddJavaContainer_ExportsTelemetryToAspire and WithJvmArgs_AppliesToAContainerToo resolve the container OTLP endpoint in run mode, which asks DCP for the container host name by running the real 'dcp info'. On busy Windows runners DCP's 'docker --version' probe can exceed the 25s dependency check timeout and the tests fail. Register a deterministic IDcpDependencyCheckService instead, and assert the host name DCP reports flows into OTEL_EXPORTER_OTLP_ENDPOINT. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21939Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21939" |
Tests selector40 / 101 PR test projects · 0 PR jobs, from 2 changed files. Selected PR test projects (40 / 101)
Selected PR jobs (0)none How these were chosen — grouped by what changed
🧪 show 39
🧪 Job reasonsnone Selection computed for commit |
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues remain for the test-only changes that isolate Java container environment-variable tests from live DCP probes.
No open issues 🎉
What changed in this PR
This test-only change isolates Java container environment-variable tests from live DCP probes, avoiding container-runtime timeout failures while preserving the real endpoint-resolution path.
Changes:
- Register the shared deterministic DCP service in both affected tests.
- Assert the exact telemetry endpoint using a runtime-specific host name.
- Grant Java tests access to internal test utilities.
| File | Description |
|---|---|
| tests/Aspire.Hosting.TestUtilities/Aspire.Hosting.TestUtilities.csproj | Grants Java tests access to the shared internal DCP test service. |
| tests/Aspire.Hosting.Java.Tests/AddJavaContainerTests.cs | Replaces live DCP lookups with deterministic data and strengthens the telemetry endpoint assertion. |
Review effort: Balanced
|
✅ No documentation update needed. docs_optional → test_only. Triggered signal: |
…ainers (#21957) ## Description **What failed.** On Windows runners, the Deno functional tests (`DenoTelemetryFunctionalTests.VerifyDenoAppExportsNativeTelemetryToDashboard`, `DenoFunctionalTests.VerifyDenoAppDirectExecutionWorks`, `DenoFunctionalTests.VerifyDenoAppTaskScriptWorks`) and other DCP-backed fixtures fail in `InitializeAsync` with: ``` Aspire.Hosting.DistributedApplicationException : Application orchestrator dependency check returned an error: The operation has timed out. dcp.info.os-executor Context expired, stopping process... {"Command": "C:\\Windows\\system32\\docker.exe", "Args": ["--version"]} ``` Example runs: [42157416383](https://github.com/microsoft/aspire/actions/runs/42157416383) (Hosting.JavaScript), [42076556098](https://github.com/microsoft/aspire/actions/runs/42076556098) (Hosting.JavaScript), [41910295423](https://github.com/microsoft/aspire/actions/runs/41910295423) (Hosting-2, Hosting.DotnetTool), [42088737959](https://github.com/microsoft/aspire/actions/runs/42088737959) (Hosting.Testing), [42253714471](https://github.com/microsoft/aspire/actions/runs/42253714471) (Hosting-4). **Why.** `DcpHost.EnsureDcpContainerRuntimeAsync` and `DcpExecutor.RunApplicationAsync` ran `dcp info` on every run-mode start. `dcp info` probes the container runtime CLI (`docker --version`). When that CLI is slow or hung on a busy Windows runner, DCP spends about 10s on the probe and then about 15s stopping it. That lands right on Aspire's 25s `DependencyCheckTimeout`, so startup fails even for apps with no containers. The Deno fixtures only have executables, so the container-runtime result is never used. **How it's fixed.** - `IDcpDependencyCheckService` gains `GetDcpVersionAsync`. It validates the DCP install and minimum version with `dcp version`, which prints the same `{"version": ...}` JSON but doesn't probe the container runtime. `dcp version` is supported back to the minimum DCP 0.23.3. `DcpDependencyCheck` keeps one nullable `_dcpInfo` with the last successful result: a version check reuses any cached result, and a full-info check reuses it only when it includes container status. A failed or cancelled probe leaves the cache unchanged, so it is retried. - Empty stdout or a JSON `null` from DCP now throws a dependency-check exception instead of returning null, so startup can't continue without a validated DCP version. - `DcpHost` and `DcpExecutor` use `GetDcpVersionAsync` when the model has no container resources. Apps with containers keep the full `dcp info` check and still fail loudly when the runtime is unhealthy. - `HostUrl` and `ContainerNetworkEndpointProvisioner` still call `dcp info`, but only for container networks, so executables never reach them. This is a product-layer fix, so it covers every executable-only app and test fixture, not just Deno. It complements the test-only approaches in #21928 and #21939, which avoid DCP in tests that don't need it. Neither of those covers the Deno tests. Fixes #21771 Fixes #21848 Related: #20850 (same Deno fixture; that report is a different DCP SQLite startup crash and is not addressed here) ## Confidence High that this removes the `docker --version` timeout for executable-only apps. On that path the failing call is no longer made, as the paired unit tests and the real-DCP repro with a hung `docker` show. The main limit is volume: the probe signature has only 4 CI hits in 14 days, and none since 11-06 21:06Z, so CI can't yet confirm the fix by itself. Most of the Deno failures in the 14-day window have other causes (see below). Those stopped after the DCP 0.27.3 bump and are not addressed here. **How we stop this recurring:** startup for apps without containers no longer depends on how responsive the container-runtime CLI is. The new `DcpHost` and `DcpExecutor` tests fail if a no-container path starts calling `dcp info` again. ## Failure stats (14d, attempt 1) Window: 10-26 10:00Z to 11-09 10:00Z, `ci.yml`, attempt 1 only. The denominator is the **757 runs where a `Hosting.JavaScript` job executed and finished (157 main + 600 PR)**. One PR run is excluded as PR-caused: every Node and Deno test failed on both OSes, and the reruns kept failing. | test | failed runs (main + PR) | rate | passed on rerun | |---|---|---|---| | `DenoTelemetryFunctionalTests.VerifyDenoAppExportsNativeTelemetryToDashboard` | 25 (6 + 19) | 3.3% | 14 | | `DenoFunctionalTests.VerifyDenoAppDirectExecutionWorks` | 25 (7 + 18) | 3.3% | 11 | | `DenoFunctionalTests.VerifyDenoAppTaskScriptWorks` | 25 (7 + 18) | 3.3% | 11 | 46 runs had at least one Deno failure (11 main, 35 PR). By cause: - **DCP probe timeout** (`dependency check returned an error` / `docker.exe --version`), which this PR fixes: **4 runs (2 main, 2 PR)**, 11-05 17:38Z to 11-06 21:06Z, all three tests each time. Both PR runs passed on rerun; the main runs weren't rerun. - 2-minute Polly timeout in fixture init: 26 runs, last 11-03 00:36Z. - `Service … should have valid address at this point`: 16 runs, last 11-02 15:08Z. The last two causes stopped after the DCP 0.27.3 bump (#21706, merged 11-03 05:38Z). Since 11-03, **every Deno failure has been the probe signature (4/334 runs, 1.2%)**. The tracker #21600 counts 20/17/17 over all 1,173 decided `ci.yml` runs. Its numerator and denominator differ from this job-level count. The same probe signature also hits other executable-only fixtures (Hosting.Testing, Hosting.DotnetTool, Hosting-N); see #21771. Those are not counted here. Runs: [42076556098](https://github.com/microsoft/aspire/actions/runs/42076556098) (main, probe), [42157416383](https://github.com/microsoft/aspire/actions/runs/42157416383) (main, probe), [42246915065](https://github.com/microsoft/aspire/actions/runs/42246915065) (PR, probe), [42293916898](https://github.com/microsoft/aspire/actions/runs/42293916898) (PR, probe), [41554110222](https://github.com/microsoft/aspire/actions/runs/41554110222) (main, older Polly-timeout cause). - **Still hits main?** Last main hit was 11-06 03:34Z (probe). There have been no hits in the 68 runs since 11-06 21:06Z (13 of them on main). - **Local before/after:** the no-container regression tests fail without the fix and pass 3/3 with it. With a hung `docker` on PATH, all 3 real Deno tests fail fixture init without the fix. With the fix, `VerifyDenoAppDirectExecutionWorks` and `VerifyDenoAppTaskScriptWorks` pass 3/3, and the telemetry test gets past fixture init (details under Evidence). - **This PR's CI:** on the current head (rebased onto main), Final Results passed on attempt 1 ([42749913164](https://github.com/microsoft/aspire/actions/runs/42749913164)), with no reruns. Every job passed, including `Hosting.JavaScript` (where the Deno tests run) and Hosting-1..6 on Windows and Ubuntu. The only red run on the head is the Copilot code-review bot, which timed out waiting for a runner; it is not a CI test job. **Verdict: MODERATE.** The fix removes the exact call behind all 4 probe-signature failures, and a deterministic local repro flips from fail to pass. But with only 4 hits in 14 days and none since 11-06, CI can't confirm it yet. ## Evidence (paired, local Linux) Root cause with real DCP 0.27.3 and a fake `docker` that hangs: | command | result | |---|---| | `dcp info --container-runtime docker` | 25s, `Context expired, stopping process... docker --version` (the CI signature) | | `dcp version` | 0s, `{"version":"0.27.3",...}` | Re-checked with DCP 0.27.4 (current main): `dcp version` returns only version fields, and `dcp info` always includes a `containers` block, even when `docker` fails or isn't on PATH. So `Containers != null` reliably identifies a cached `dcp info` result. Regression tests (`Aspire.Hosting.Tests`): | test | without fix | with fix | |---|---|---| | `DcpHostNotificationTests.EnsureDcpContainerRuntimeAsync_WithoutContainers_DoesNotWaitForContainerRuntimeProbe` | ❌ `dependency check returned an error: The operation has timed out` | ✅ (3/3) | | `DcpExecutorTests.RunApplicationAsync_WithoutContainers_DoesNotProbeContainerRuntime` | ❌ `GetDcpInfoAsync` was called | ✅ (3/3) | | `DcpHostNotificationTests.EnsureDcpContainerRuntimeAsync_WithContainers_ReportsContainerRuntimeProbeTimeout` | ✅ | ✅ (containers still fail loudly) | | `DcpExecutorTests.RunApplicationAsync_WithContainers_ProbesContainerRuntime` | ✅ | ✅ | | `DcpHostNotificationTests.GetDcpVersionAsync_RetriesAfterFailedCheck` | ❌ (failed probe cached as null) | ✅ | | `DcpHostNotificationTests.DependencyCheck_ThrowsWhenDcpReturnsNoInformation` (empty / `null` output, `version` and `info`) | ❌ (returned null) | ✅ | All 377 tests in `DcpHostNotificationTests` and `DcpExecutorTests` pass on the current head. The real Deno tests, with a hung `docker` on PATH and `ASPIRE_DEPENDENCY_CHECK_TIMEOUT=8` to make the race deterministic: - **Without fix:** all 3 Deno tests fail in fixture init with `Application orchestrator dependency check returned an error: The operation has timed out.` - **With fix:** both fixtures initialize, and `VerifyDenoAppDirectExecutionWorks` and `VerifyDenoAppTaskScriptWorks` pass 3/3. The telemetry test gets past fixture init. Its dashboard-query step times out in this sandbox on `main` as well, with or without the hung `docker`, so that part is environmental and unrelated. ## Checklist - Is this feature complete? - [x] Yes. Ready to ship. - [ ] No. Follow-up changes expected. - Are you including unit tests for the changes and scenario tests if relevant? - [x] Yes - [ ] No - Did you add public API? - [ ] Yes - [x] No - Does the change make any security assumptions or guarantees? - [ ] Yes - [x] No **🤖 Posted by AspireAgent (AI Agent)** --------- Co-authored-by: davidfowl <95136+davidfowl@users.noreply.github.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
What failed
Aspire.Hosting.Java.Tests.AddJavaContainerTests.AddJavaContainer_ExportsTelemetryToAspirefailed inHosting.Java (windows-latest)on attempt 1 with:Runs: 41873115695 (main, 11-04) and 42181390567 (PR, 11-06). This is rank 5 in the unowned-flakes list on #21600.
Why
The test builds a run-mode app and resolves the container's environment.
OTEL_EXPORTER_OTLP_ENDPOINTis aHostUrl, and in run modeHostUrlasksIDcpDependencyCheckServicefor the container host name. The real service runsdcp infounder the 25sDcpOptions.DependencyCheckTimeout. On busy Windows runners, DCP'sdocker --versionprobe insidedcp infosometimes takes longer than that (the shared infra cause tracked in #21771), so the test fails before it reaches its assertion.WithJvmArgs_AppliesToAContainerTooin the same class goes through the same path.The test only needs a container host name from DCP. It doesn't need DCP itself. Its old assertion also compared the env var against a second call to that same live service, so it couldn't catch
HostUrlignoring the host name DCP reports.How it's fixed
Test-only change. Both tests register a deterministic
IDcpDependencyCheckService(the existingTestDcpDependencyCheckServicefromAspire.Hosting.TestUtilities, which now grantsInternalsVisibleTotoAspire.Hosting.Java.Testslike the Go and Dotnet test projects).AddJavaContainer_ExportsTelemetryToAspirenow reportshost.wslc.internal(the runtime-specific case from #21244) and asserts the exacthttp://host.wslc.internal:4317. The test still builds the app and goes through the realHostUrlandDcpOptionspath. No timeouts, retries or assertions were relaxed.Related: #21316 (closed without merging) targeted an earlier failure of this same test, a different root cause: DCP reported an empty container host name, which produced
http://:4317. #21432 fixed that cause on main on 10-30. This PR fixes the remainingdcp infotimeout signature, which neither PR covers.Confidence
2f471dc6,DcpPublisher__CliPathpointing at a fakedcpwhoseinfohangs for 60s): 3/3 runs fail. Both tests fail after about 30s with the sameApplication orchestrator dependency check returned an error: The operation has timed outmessage as CI.dcp: 10/10 runs pass (14/14 tests, about 1s).HostUrlis changed to ignore DCP's host name (alwayshost.docker.internal), the new test fails withExpected: "http://host.wslc.internal:4317",Actual: "http://host.docker.internal:4317". The old assertion would still pass on any runner where DCP reportshost.docker.internal.docker --versionslowness isn't fixed here. It can still affect tests that actually start DCP, and [CI Failure] DCP dependency check times out running docker --version on Windows runners #21771 tracks it.How we stop this recurring: unit tests that only need a value from DCP (such as the container host name) should register a deterministic
IDcpDependencyCheckServiceinstead of running the realdcp info.Failure stats (14d, attempt 1)
Source: a scan of every CI (
ci.yml) run created 10-09 → 11-08 13:00Z, using attempt-1Hosting.Javajobs and their logs.Aspire.Hosting.Java.Tests.AddJavaContainerTests.AddJavaContainer_ExportsTelemetryToAspirefailing withApplication orchestrator dependency check returned an error: The operation has timed out ... docker.exe --version(Windows).Hosting.Javajob ran (0.6%). PR: 1/471 (0.2%). Including the siblingWithJvmArgs_AppliesToAContainerToo, which has the same signature and is fixed here, PR is 2/471 (0.4%).Strings differ(WSLc and empty host name), fixed by Fix Java container telemetry test for runtime-specific hostnames #21254 and Fall back to default container host name when DCP reports an empty one #21432. None has appeared since Fall back to default container host name when DCP reports an empty one #21432 merged on 10-30 15:51Z. One more hit, 41194713391 (DCPCliPathvalidation on both OSes), is branch-caused and excluded.dcp info. After the fix, 10/10 passed with a hangingdcp infoand 20/20 with real DCP. The mutation check fails as expected.Hosting.Javaon Windows and Ubuntu.Checklist
🤖 Posted by AspireAgent (AI Agent)