fix(aliases): retry ReplaceAliases when a concurrent writer wins the unique-index race - #268
Open
jordanfelle wants to merge 2 commits into
Open
jordanfelle wants to merge 2 commits into
jordanfelle wants to merge 2 commits into
Conversation
…unique-index race
ProviderAliasRepository.ReplaceAliases is DELETE-then-INSERT in a READ COMMITTED transaction.
Two writers replacing the same entity's aliases at once both DELETE nothing, one INSERTs and
commits, and the other's INSERT violates IX_ProviderAliasIndex_Unique. In practice a bulk
author edit (2,567 authors moved between root folders) failed with a 500:
PostgresException 23505 duplicate key value violates "IX_ProviderAliasIndex_Unique"
at InsertMany <- AuthorService.RefreshAuthorProviderAliases <- UpdateAuthors <- AuthorEditorController.SaveAll
Within one call the aliases are already de-duplicated after normalisation and the DELETE runs
first in the same transaction, so a single caller cannot collide with itself - the collision
is between writers. Reproduced the exact interleaving on a scratch Postgres table (writer B's
INSERT fails; re-running DELETE+INSERT afterwards succeeds).
Retry the replace (up to 3 attempts, short growing pause) when the failure is a unique
violation (Npgsql SqlState 23505, or SQLite "UNIQUE constraint failed", also when wrapped), so
the loser's DELETE now sees the winner's committed rows and replaces them: last writer wins.
Fresh ProviderAlias instances per attempt. Other errors are not retried and a persistent
violation is rethrown after the last attempt. Book aliases use the same method and get the
same protection.
Tests: retry on Postgres/SQLite unique violation then succeed; wrapped exception found; other
errors and non-unique SQLite constraint errors are not retried; a persistent violation is
rethrown after 3 attempts.
… real SQLite database Adversarial review of Chaptarr#268: nothing checked that ReplaceAliases actually goes through RetryOnUniqueViolation, or that copies are inserted. Make ReplaceAliasesOnce internal virtual so a test subclass can fail the first attempt (wiring test fails if the retry is bypassed), and add a real-SQLite test that replaces an entity's rows twice without colliding and leaves another entity's aliases untouched.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A bulk author edit (root folders changed for 2,567 authors) failed with a 500:
ProviderAliasRepository.ReplaceAliasesis DELETE-then-INSERT in a READ COMMITTED transaction. Within one call the aliases are already de-duplicated after normalisation, and the DELETE runs first in the same transaction, so a single caller cannot collide with itself. Two writers replacing the same entity at once can: both DELETE nothing, one INSERTs and commits, the other's INSERT violates the unique index. A bulk edit updates thousands of authors and each update fires events whose handlers refresh the same aliases, so this is easy to hit.Reproduced the exact interleaving on a scratch Postgres table: writer B's INSERT fails with the same error, and re-running DELETE+INSERT afterwards succeeds.
Change
Retry the replace (up to 3 attempts, short growing pause) when the failure is a unique violation (Npgsql
SqlState 23505, or SQLiteUNIQUE constraint failed, including when wrapped). The loser's DELETE then sees the winner's committed rows and replaces them: last writer wins. FreshProviderAliasinstances are built per attempt. Other errors are not retried and a persistent violation is rethrown after the last attempt. Book aliases go through the same method and get the same protection.Tests
ProviderAliasRepositoryRetryFixture(9): retry then succeed on Postgres and SQLite violations; wrapped exception detected; other DB errors and non-unique SQLite constraint errors are not retried; a persistent violation is rethrown after 3 attempts. FullChaptarr.Core.Test: 3044 passed.Not covered
The retry wiring inside
ReplaceAliasesitself is not exercised against a real database in the unit tests (the helper is). The aliases of authors after the failing one in a bulk edit were left un-refreshed by the original failure; a normal refresh rebuilds them.Follow-up from adversarial review
ReplaceAliasesOnceis nowinternal virtualas the test seam.