Skip to content

Serialize container registry logins to avoid credential overwrite races - #21458

Open
David Fowler (davidfowl) wants to merge 3 commits into
mainfrom
agent/serialize-registry-logins
Open

David Fowler (davidfowl) wants to merge 3 commits into
mainfrom
agent/serialize-registry-logins

Conversation

@davidfowl

@davidfowl David Fowler (davidfowl) commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Description

Fixes a race where concurrent Azure Container Registry login steps overwrite each other's Docker credentials, so a later image push fails with UNAUTHORIZED.

Fixes #21090

Root cause

  • Each AzureContainerRegistryResource registers its own login-to-acr-{name} pipeline step. These steps have no ordering between them, so with two or more registries they run concurrently.
  • Each step calls AcrLoginService.LoginAsync, which calls IContainerRuntime.LoginToRegistryAsync.
  • ContainerRuntimeBase.LoginToRegistryAsync starts docker login (or podman login) with --password-stdin and nothing coordinates the calls. The container runtime is a singleton.
  • Every login rewrites the runtime's shared credential config file (~/.docker/config.json or the credential store). Two overlapping logins can lose one registry's credentials, and the later push to that registry is rejected. The reporter's repro (https://github.com/mip1983/AspireACRConcurrency) shows this timeline: two overlapping logins, then an UNAUTHORIZED push.

Fix

ContainerRuntimeBase now serializes LoginToRegistryAsync per runtime instance with a SemaphoreSlim:

  • Waiting for the lock honors the cancellation token.
  • The lock is held until the login process is disposed.
  • The original body moved unchanged into a private LoginToRegistryCoreAsync.

Login steps still run in the pipeline as before. Only the external login command is no longer run concurrently.

Tests

Two tests added to ContainerRuntimeBaseTests. They use the shared TestProcessRunner with EnqueuePending, so each login process stays open until the test completes it. The overlap is deterministic and needs no sleeps:

  • LoginToRegistryAsync_SerializesOverlappingLoginsToDifferentRegistries: two overlapping logins to different registries. The second login must not start until the first finishes and is disposed. The max concurrency must be 1.
  • LoginToRegistryAsync_CanceledWhileWaitingDoesNotStartLogin: a login cancelled while it waits must throw and must never start a process.

Before/after (local)

Result
Before (fix stashed) Both new tests fail: Assert.Single() Failure: The collection contained 2 items, meaning two login processes ran concurrently.
After New tests 0/25 failures across repeated runs. ContainerRuntimeBaseTests and DockerContainerRuntimeTests: 41/41 passed.

Context

This came from the slice B fallback. A scan of PR ci.yml first attempts on Oct 16–30 (541 runs whose latest attempt was 1) found no qualifying flaky test in the component or macOS jobs. Component failures were MongoDB only, which is owned elsewhere. All recurring macOS failures belong to the already-tracked macOS test-host hang family. #21090 was open, unassigned and had no PR.

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

🤖 Posted by AspireAgent (AI Agent)

Concurrent login-to-acr-* pipeline steps ran 'docker login' at the same time,
and the logins could overwrite each other's entries in the shared credential
config file, causing UNAUTHORIZED pushes. ContainerRuntimeBase now serializes
LoginToRegistryAsync per runtime instance.

Fixes #21090

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21458

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21458"

@aspire-repo-bot
aspire-repo-bot Bot requested a balanced review from Copilot October 30, 2026 14:31
@github-actions github-actions Bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Oct 30, 2026
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The tests duplicate the shared TestProcessRunner with a private runner, contrary to the test guidelines’ shared-service requirement.

1 open issue
What changed in this PR

This PR addresses registry credential overwrite races by serializing container login commands within each runtime instance, without serializing image builds or pushes.

Changes:

  • Adds a cancellation-aware semaphore held through login process disposal.
  • Moves the existing login implementation into a private core method.
  • Adds deterministic tests for overlapping logins and cancellation while waiting.
File Description
tests/​Aspire.Hosting.Tests/​Publishing/​ContainerRuntimeBaseTests.cs Adds gated-process tests for login serialization and canceled waiters.
src/​Aspire.Hosting/​Publishing/​ContainerRuntimeBase.cs Serializes registry logins with a per-instance semaphore.

Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

This comment has been minimized.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the registry-login serialization change.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

@github-actions

Copy link
Copy Markdown
Contributor

Retrying the failed CI jobs for this pull request from the CI run attempt. The rerun is being tracked in the rerun attempt.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@davidfowl David Fowler (davidfowl) added the agent:working An AI agent is actively working on this issue or PR label Nov 6, 2026 — with Copilot SWE Agent

Copy link
Copy Markdown
Collaborator Author

Merged main to resolve the conflict with #21009. The only conflict was a using directive in ContainerRuntimeBaseTests.cs (kept both Aspire.Hosting.Tests.Utils and Aspire.Shared); ContainerRuntimeBase.cs auto-merged, keeping #21009's WSLc changes and this PR's registry-login lock unchanged. No assertions changed.

Locally: all 28 ContainerRuntimeBaseTests pass; with the lock removed, both login regression tests fail.

🤖 Posted by AspireAgent (AI Agent)

@github-actions

github-actions Bot commented Nov 6, 2026

Copy link
Copy Markdown
Contributor

Tests selector

52 / 101 PR test projects · 4 PR jobs, from 2 changed files.

Selected PR test projects (52 / 101)

Aspire.Cli.EndToEnd.Tests, Aspire.Hosting.Analyzers.Tests, Aspire.Hosting.Azure.EntraId.Tests, Aspire.Hosting.Azure.Kubernetes.Tests, Aspire.Hosting.Azure.Kusto.Tests, Aspire.Hosting.Azure.Provisioning.Tests, Aspire.Hosting.Azure.Tests, Aspire.Hosting.Blazor.Tests, Aspire.Hosting.Browsers.Tests, Aspire.Hosting.CodeGeneration.Go.Tests, Aspire.Hosting.CodeGeneration.Java.Tests, Aspire.Hosting.CodeGeneration.Python.Tests, Aspire.Hosting.CodeGeneration.Rust.Tests, Aspire.Hosting.CodeGeneration.TypeScript.Tests, Aspire.Hosting.Containers.Tests, Aspire.Hosting.DevTunnels.Tests, Aspire.Hosting.Docker.Tests, Aspire.Hosting.Dotnet.Tests, Aspire.Hosting.DotnetTool.Tests, Aspire.Hosting.EntityFrameworkCore.Tests, Aspire.Hosting.Foundry.Tests, Aspire.Hosting.Garnet.Tests, Aspire.Hosting.Go.Tests, Aspire.Hosting.Java.Tests, Aspire.Hosting.JavaScript.Tests, Aspire.Hosting.Kafka.Tests, Aspire.Hosting.Keycloak.Tests, Aspire.Hosting.Kubernetes.Tests, Aspire.Hosting.Maui.Tests, Aspire.Hosting.Milvus.Tests, Aspire.Hosting.MongoDB.Tests, Aspire.Hosting.MySql.Tests, Aspire.Hosting.Nats.Tests, Aspire.Hosting.OpenAI.Tests, Aspire.Hosting.Oracle.Tests, Aspire.Hosting.Orleans.Tests, Aspire.Hosting.PostgreSQL.Tests, Aspire.Hosting.Python.Tests, Aspire.Hosting.Qdrant.Tests, Aspire.Hosting.RabbitMQ.Tests, Aspire.Hosting.Radius.Tests, Aspire.Hosting.Redis.Tests, Aspire.Hosting.RemoteHost.Tests, Aspire.Hosting.Rust.Tests, Aspire.Hosting.Seq.Tests, Aspire.Hosting.SqlServer.Tests, Aspire.Hosting.Testing.Tests, Aspire.Hosting.Tests, Aspire.Hosting.Valkey.Tests, Aspire.Hosting.Yarp.Tests, Aspire.Playground.Tests, Aspire.Templates.Tests

Selected PR jobs (4)

cli-starter-validation, extension-e2e, polyglot, typescript-api-compat


How these were chosen — grouped by what changed

⚠️ 46 of the 52 selected test projects come from a single change — src/Aspire.Hosting/Publishing/ContainerRuntimeBase.cs.

🔧 src/Aspire.Hosting/Publishing/ContainerRuntimeBase.cs (changed source)
→ 46 via the project graph

show 46

Aspire.Hosting.Analyzers.Tests (2 hops), Aspire.Hosting.Azure.EntraId.Tests (2 hops), Aspire.Hosting.Azure.Kubernetes.Tests (2 hops), Aspire.Hosting.Azure.Kusto.Tests (2 hops), Aspire.Hosting.Azure.Provisioning.Tests (3 hops), Aspire.Hosting.Azure.Tests, Aspire.Hosting.Browsers.Tests (2 hops), Aspire.Hosting.CodeGeneration.Go.Tests, Aspire.Hosting.CodeGeneration.Java.Tests, Aspire.Hosting.CodeGeneration.Python.Tests, Aspire.Hosting.CodeGeneration.Rust.Tests, Aspire.Hosting.CodeGeneration.TypeScript.Tests, Aspire.Hosting.Containers.Tests (2 hops), Aspire.Hosting.DevTunnels.Tests (2 hops), Aspire.Hosting.Docker.Tests (2 hops), Aspire.Hosting.DotnetTool.Tests (2 hops), Aspire.Hosting.EntityFrameworkCore.Tests (2 hops), Aspire.Hosting.Foundry.Tests (2 hops), Aspire.Hosting.Garnet.Tests (2 hops), Aspire.Hosting.Go.Tests (2 hops), Aspire.Hosting.Java.Tests (2 hops), Aspire.Hosting.JavaScript.Tests (2 hops), Aspire.Hosting.Kafka.Tests (2 hops), Aspire.Hosting.Keycloak.Tests (2 hops), Aspire.Hosting.Kubernetes.Tests (2 hops), Aspire.Hosting.Maui.Tests, Aspire.Hosting.Milvus.Tests (2 hops), Aspire.Hosting.MongoDB.Tests (2 hops), Aspire.Hosting.MySql.Tests (2 hops), Aspire.Hosting.Nats.Tests (2 hops), Aspire.Hosting.OpenAI.Tests (2 hops), Aspire.Hosting.Oracle.Tests (2 hops), Aspire.Hosting.Orleans.Tests (2 hops), Aspire.Hosting.PostgreSQL.Tests (2 hops), Aspire.Hosting.Python.Tests (2 hops), Aspire.Hosting.Qdrant.Tests (2 hops), Aspire.Hosting.RabbitMQ.Tests (2 hops), Aspire.Hosting.Redis.Tests (2 hops), Aspire.Hosting.RemoteHost.Tests, Aspire.Hosting.Rust.Tests (2 hops), Aspire.Hosting.Seq.Tests (2 hops), Aspire.Hosting.SqlServer.Tests (2 hops), Aspire.Hosting.Testing.Tests (2 hops), Aspire.Hosting.Valkey.Tests (2 hops), Aspire.Hosting.Yarp.Tests (2 hops), Aspire.Playground.Tests

🧪 tests/Aspire.Hosting.Tests/Publishing/ContainerRuntimeBaseTests.cs (changed test)
→ 1 directly: Aspire.Hosting.Tests
→ 3 via the project graph: Aspire.Hosting.Blazor.Tests, Aspire.Hosting.Dotnet.Tests, Aspire.Hosting.Radius.Tests

📦 affected project Aspire.Hosting.Dotnet
→ 2 tests: Aspire.Cli.EndToEnd.Tests, Aspire.Templates.Tests

📦 affected project Aspire.Hosting
→ 1 test: Aspire.Cli.EndToEnd.Tests

Job reasons

Job Triggered by
cli-starter-validation affected project Aspire.Hosting.AppHost
extension-e2e • src/Aspire.Hosting/Publishing/ContainerRuntimeBase.cs
• affected project Aspire.Hosting
polyglot • affected project Aspire.Hosting
• affected project Aspire.Hosting.Azure.Provisioning
typescript-api-compat affected project Aspire.Hosting

Selection computed for commit 49d8a7d.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the change that serializes registry logins per container runtime instance.

No open issues 🎉

Review effort: Balanced

@davidfowl David Fowler (davidfowl) removed the agent:working An AI agent is actively working on this issue or PR label Nov 6, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area-deployment needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Azure Container Registry: concurrent login steps can overwrite Docker credentials and cause unauthorized pushes

2 participants