Repository navigation
Serialize container registry logins to avoid credential overwrite races - #21458
David Fowler (davidfowl) wants to merge 3 commits into
Conversation
Concurrent login-to-acr-* pipeline steps ran 'docker login' at the same time, and the logins could overwrite each other's entries in the shared credential config file, causing UNAUTHORIZED pushes. ContainerRuntimeBase now serializes LoginToRegistryAsync per runtime instance. Fixes #21090 Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
🚀 Dogfood this PR with:
curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21458Or
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21458" |
This comment has been minimized.
This comment has been minimized.
There was a problem hiding this comment.
🟡 Changes recommended
The tests duplicate the shared TestProcessRunner with a private runner, contrary to the test guidelines’ shared-service requirement.
1 open issue
What changed in this PR
This PR addresses registry credential overwrite races by serializing container login commands within each runtime instance, without serializing image builds or pushes.
Changes:
- Adds a cancellation-aware semaphore held through login process disposal.
- Moves the existing login implementation into a private core method.
- Adds deterministic tests for overlapping logins and cancellation while waiting.
| File | Description |
|---|---|
| tests/Aspire.Hosting.Tests/Publishing/ContainerRuntimeBaseTests.cs | Adds gated-process tests for login serialization and canceled waiters. |
| src/Aspire.Hosting/Publishing/ContainerRuntimeBase.cs | Serializes registry logins with a per-instance semaphore. |
Review effort: Balanced
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This comment has been minimized.
This comment has been minimized.
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
|
Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt. |
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
|
Merged Locally: all 28 🤖 Posted by AspireAgent (AI Agent) |
Tests selector52 / 101 PR test projects · 4 PR jobs, from 2 changed files. Selected PR test projects (52 / 101)
Selected PR jobs (4)
How these were chosen — grouped by what changed
🔧 show 46
🧪 📦 affected project 📦 affected project Job reasons
Selection computed for commit |

Description
Fixes a race where concurrent Azure Container Registry login steps overwrite each other's Docker credentials, so a later image push fails with
UNAUTHORIZED.Fixes #21090
Root cause
AzureContainerRegistryResourceregisters its ownlogin-to-acr-{name}pipeline step. These steps have no ordering between them, so with two or more registries they run concurrently.AcrLoginService.LoginAsync, which callsIContainerRuntime.LoginToRegistryAsync.ContainerRuntimeBase.LoginToRegistryAsyncstartsdocker login(orpodman login) with--password-stdinand nothing coordinates the calls. The container runtime is a singleton.~/.docker/config.jsonor the credential store). Two overlapping logins can lose one registry's credentials, and the later push to that registry is rejected. The reporter's repro (https://github.com/mip1983/AspireACRConcurrency) shows this timeline: two overlapping logins, then an UNAUTHORIZED push.Fix
ContainerRuntimeBasenow serializesLoginToRegistryAsyncper runtime instance with aSemaphoreSlim:LoginToRegistryCoreAsync.Login steps still run in the pipeline as before. Only the external login command is no longer run concurrently.
Tests
Two tests added to
ContainerRuntimeBaseTests. They use the sharedTestProcessRunnerwithEnqueuePending, so each login process stays open until the test completes it. The overlap is deterministic and needs no sleeps:LoginToRegistryAsync_SerializesOverlappingLoginsToDifferentRegistries: two overlapping logins to different registries. The second login must not start until the first finishes and is disposed. The max concurrency must be 1.LoginToRegistryAsync_CanceledWhileWaitingDoesNotStartLogin: a login cancelled while it waits must throw and must never start a process.Before/after (local)
Assert.Single() Failure: The collection contained 2 items, meaning two login processes ran concurrently.ContainerRuntimeBaseTestsandDockerContainerRuntimeTests: 41/41 passed.Context
This came from the slice B fallback. A scan of PR
ci.ymlfirst attempts on Oct 16–30 (541 runs whose latest attempt was 1) found no qualifying flaky test in the component or macOS jobs. Component failures were MongoDB only, which is owned elsewhere. All recurring macOS failures belong to the already-tracked macOS test-host hang family. #21090 was open, unassigned and had no PR.Checklist
<remarks />and<code />elements on your triple slash comments?🤖 Posted by AspireAgent (AI Agent)