Skip to content

Wait for a writable MongoDB single-member primary - #21439

Closed
David Fowler (davidfowl) wants to merge 1 commit into
mainfrom
aspireagent/fix-mongodb-single-member-writable-primary
Closed

David Fowler (davidfowl) wants to merge 1 commit into
mainfrom
aspireagent/fix-mongodb-single-member-writable-primary

Conversation

@davidfowl

Copy link
Copy Markdown
Collaborator

Description

Fixes an intermittent MongoNotPrimaryException : Server returned not primary error in SingleMemberReplicaSetFunctionalTests on Ubuntu (TransactionsAndChangeStreamsWorkWithoutADeveloperCertificate and TransactionsAndChangeStreamsWorkWithTls). The same race can affect real AppHosts that use WithReplicaSet().

Root cause. MongoDBSingleMemberReplicaSet treated the resource as ready once replSetGetStatus.myState == 1. A newly elected primary reports PRIMARY while it is still in drain mode, and it rejects writes with NotWritablePrimary until it logs "Transition to primary complete; database writes are now permitted". The resource could become healthy inside that window, so the test's first write failed. In the CI failure, replSetInitiate ran at 06:57:31.520, the "is the primary" message was logged at 31.550, orders turned healthy and the next write failed with not-primary.

Fix. Initialization and the health check now use hello. Readiness requires setName to match, isWritablePrimary == true and exactly one host. The configuration validation and initiation flow are unchanged. No assertions were weakened and no sleeps or retries were added.

Evidence

  • Native repro (mongo:9.0, --replSet, polling right after replSetInitiate):
    • The old predicate saw myState: 1 with isWritablePrimary: false, and an immediate createCollection returned NotWritablePrimary in 3/3 runs.
    • With the hello gate, PRIMARY was reported at 2–5 ms and writable at 11–27 ms, and the write succeeded in 5/5 runs.
  • New regression test InitializationWaitsUntilThePrimaryAcceptsWrites runs initialization, then writes immediately:
    • Before the fix: failed 3/3 locally with MongoNotPrimaryException.
    • After the fix: passed 3/3.
  • Unit tests ReadinessRequiresTheSingleMemberWritablePrimary and ReadinessRequiresAReplicaSetMember cover the predicate, including the drain-mode case (isWritablePrimary: false).
  • Frequency (main ci.yml, Oct 16–30):

Limitations

  • Locally, without a dev cert or a TLS-capable Docker setup, MongoExpressConnectsWithTls, TransactionsAndChangeStreamsWorkWithTls and DataAndReplicaSetIdentitySurviveNewAppHostsAndPorts fail both with and without this change. CI covers them.

Checklist

  • Is this feature complete?
    • Yes. Ready to ship.
    • No. Follow-up changes expected.
  • Are you including unit tests for the changes and scenario tests if relevant?
    • Yes
    • No
  • Did you add public API?
    • Yes
      • If yes, did you have an API Review for it?
        • Yes
        • No
      • Did you add <remarks /> and <code /> elements on your triple slash comments?
        • Yes
        • No
    • No
  • Does the change make any security assumptions or guarantees?
    • Yes
      • If yes, have you done a threat model and had a security review?
        • Yes
        • No
    • No

🤖 Posted by AspireAgent (AI Agent)

A newly elected primary reports replSetGetStatus myState 1 while it is still
in drain mode and rejects writes with NotWritablePrimary. Gate single-member
initialization and health on hello.isWritablePrimary instead.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 21439

Or

  • Run remotely in PowerShell:
iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 21439"

@github-actions github-actions Bot added the needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners label Oct 30, 2026
@aspire-repo-bot
aspire-repo-bot Bot requested a balanced review from Copilot October 30, 2026 09:05
@github-actions

Copy link
Copy Markdown
Contributor

Tests selector

4 / 100 PR test projects · 2 PR jobs, from 4 changed files.

Selected PR test projects (4 / 100)

Aspire.Cli.EndToEnd.Tests, Aspire.Hosting.MongoDB.Tests, Aspire.Hosting.Radius.Tests, Aspire.Playground.Tests

Selected PR jobs (2)

polyglot, typescript-api-compat


How these were chosen — grouped by what changed

🔧 src/Aspire.Hosting.MongoDB/MongoDBBuilderExtensions.cs (changed source)
→ 1 directly: Aspire.Hosting.MongoDB.Tests
→ 2 via the project graph: Aspire.Hosting.Radius.Tests, Aspire.Playground.Tests (2 hops)

📦 affected project Aspire.Hosting.MongoDB
→ 1 test: Aspire.Cli.EndToEnd.Tests

🔧 src/Aspire.Hosting.MongoDB/MongoDBSingleMemberReplicaSet.cs (changed source)
→ 1 directly: Aspire.Hosting.MongoDB.Tests

🧪 tests/Aspire.Hosting.MongoDB.Tests/SingleMemberReplicaSetFunctionalTests.cs (changed test)
→ 1 directly: Aspire.Hosting.MongoDB.Tests

🧪 tests/Aspire.Hosting.MongoDB.Tests/SingleMemberReplicaSetTests.cs (changed test)
→ 1 directly: Aspire.Hosting.MongoDB.Tests

Job reasons

Job Triggered by
polyglot affected project Aspire.Hosting.MongoDB
typescript-api-compat affected project Aspire.Hosting.MongoDB

Selection computed for commit b0d58a4.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the MongoDB replica-set readiness change.

No open issues 🎉

What changed in this PR

Fixes a readiness race in Aspire’s single-member MongoDB replica sets that could allow writes before the primary was writable.

Changes:

  • Uses hello for initialization and health checks, requiring the expected replica-set name, a writable primary, and exactly one host.
  • Updates unit coverage for drain mode and missing replica-set metadata.
  • Adds a functional regression test that writes immediately after initialization.
File Description
tests/​Aspire.Hosting.MongoDB.Tests/​SingleMemberReplicaSetTests.cs Tests the writable-primary readiness predicate.
tests/​Aspire.Hosting.MongoDB.Tests/​SingleMemberReplicaSetFunctionalTests.cs Adds an immediate-write initialization regression test.
src/​Aspire.Hosting.MongoDB/​MongoDBSingleMemberReplicaSet.cs Waits for writable-primary readiness using hello.
src/​Aspire.Hosting.MongoDB/​MongoDBBuilderExtensions.cs Applies the same readiness predicate to health checks.

Review effort: Balanced

Copy link
Copy Markdown
Collaborator Author

Closing in favor of #21638, which makes the same product fix: init and the health check both use hello and require isWritablePrimary, a matching setName and exactly one host. #21638 also covers more:

  • Deterministic unit tests with a fake draining member, for both initialization and the health check. This PR tests only initialization, using a container.
  • Updated README and XML docs for the new readiness guarantee.
  • Fixes #21637, the tracking issue for this flake.

#21638 is also the PR currently going through paired CI validation.

🤖 Posted by AspireAgent (AI Agent)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

flaky-test needs-area-label An area label is needed to ensure this gets routed to the appropriate area owners

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants