Conversation
Racing puts to a new item must all succeed unless a put's own condition fails against the item before it, and their ReturnValues ALL_OLD images must form one chain, as on Amazon DynamoDB. The tests cover hash and range tables, a condition, a GSI table, and a stream table. Assisted-by: pi claude-opus-5-5
A PutItem takes the transactional path when it has a condition, ReturnValues ALL_OLD, ReturnConsumedCapacity, an index, a vector index, or a stream. BatchWriteItem puts take the same path on the same tables, and with ReturnConsumedCapacity INDEXES. On that path a put inserts a missing item with ON CONFLICT DO NOTHING. When a concurrent writer created the item first, the put returned ConditionalCheckFailed, even with no condition. BatchWriteItem returned it too, which Amazon DynamoDB never does. The put now re-reads the winner's row, checks its condition against it, and overwrites it, as UpdateItem already does. ALL_OLD, the index updates, and the stream record get the winner as the old image. If the winner was deleted before the re-read, the put retries the insert, up to 5 inserts. Storage tests hold the competing create open against a live PostgreSQL, and CI runs them. Assisted-by: pi claude-opus-5-5
The GSI and LSI tests give each put its own index key and check that only the final item's entry stays in the index. The stream tests check every round in sequence order: each record's old image is the record before it. New tests race BatchWriteItem puts on the stream table, and PutItem calls with ReturnConsumedCapacity. Assisted-by: pi claude-opus-5-5
Two storage tests park the put's insert behind a trigger while an outside transaction commits a winner and a second one locks it. The winner is then deleted before the put re-reads it. In one test the put retries its insert and creates the item. In the other it loses five inserts and returns an internal error, with nothing written. The create race tests now wait for a backend that the creator blocks, not for any lock waiter. Assisted-by: pi claude-opus-5-5
The high-level design says that a missing item has no row to lock, and how a PutItem or UpdateItem that loses the insert orders itself after the winner. Two comments in the put code now match what the code does. Assisted-by: pi claude-opus-5-5
The deleted-winner and give-up tests now run on a hash table and on a hash and range table, so the retry arm and the bound of the range branch have tests too. When a driver step fails, the driver opens the gate, and the test reports what the put returned instead of only that no backend waited. Assisted-by: pi claude-opus-5-5
The module doc of the storage tests still said two. Assisted-by: pi claude-opus-5-5
On MongoDB, a put with no condition on a table with no index and no stream returns HTTP 500 when it loses the create race, and the MongoDB write-race fix repairs it. That fix lands as its own PR. Until it is on main, the three affected tests are marked xfail when the MongoDB test runner runs them. The marker is not strict, so the tests also pass once the fix is in. Remove the marker then. Assisted-by: pi claude-opus-5-5
yesyayen
marked this pull request as ready for review
October 5, 2026 22:31
yesyayen
requested review from
LeeroyHannigan,
amrith,
c33howard,
jcshepherd,
pdf-amzn and
robinnsc
as code owners
October 5, 2026 22:31
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
On PostgreSQL, a PutItem that lost the race to create a new item returned
ConditionalCheckFailedException, even when it had no condition. It now orders itself after the winner, as UpdateItem already does.put_item_impl()incrates/storage-postgres/src/data/put_item.rs: when theINSERT ... ON CONFLICT DO NOTHINGof a missing item affects no row, the put re-reads the rowFOR UPDATE, checks its condition against the winner, and overwrites it. ReturnValues ALL_OLD returns the winner, and the stream record and index maintenance use it as the old image. If the winner was deleted before the re-read, the insert is retried, up to 5 inserts, the same bound as UpdateItem.docs/design/02-high-level-design.md.SQLite runs one writer at a time, so it does not have this bug. On MongoDB the losing put gets HTTP 500 instead, and a separate PR fixes that.
Why
Found while checking TransactGetItems isolation: 8 clients put the same new key at once. Measured on
c178814:attribute_not_exists(zz), which every racer passes, hash tableattribute_not_exists(pk)(control)On Amazon DynamoDB the ALL_OLD images of one round form a single chain, from no item to the final one, so the puts were applied one after another.
Fixes: n/a, found by a concurrency probe, no issue filed
Result
test_put_item_create_race.py(9 tests) fails 8 of 9 on main. It passes 9 of 9 on this branch in every run, on SQLite, and on MongoDB with its fix. 7 of the 8 new PostgreSQL storage tests fail on main, and all 8 pass here. Two mutants, one that drops the old image for the index updates and one that drops it for the stream record, each fail the matching tests.cargo test --release --workspace: 1,230 passed.Testing done
tests/test_put_item_create_race.py(new, dual-target): 8 clients put the same new key, 15 rounds per test. Shapes: ReturnValues ALL_OLD on hash and hash-range tables, ReturnConsumedCapacity, a condition every racer passes, a GSI table, an LSI table, a stream table, BatchWriteItem on the stream table, and theattribute_not_exists(pk)control. Every put must succeed unless its own condition fails, and the ALL_OLD images must form one chain. The GSI and LSI must hold only the final item's entry. For each key, the stream must hold one INSERT and then MODIFY records in sequence order, each with the record before it as its old image.crates/storage-postgres/tests/put_create_race.rs(new, needsEXTENDDB_TEST_PG_CONNECTION_STRING): an outside transaction holds an uncommitted insert of the item, and the test commits it once the put waits on it. The put must overwrite the winner and return it as ALL_OLD on hash and hash-range tables, pass a condition that holds for the winner, and fail a condition that the winner breaks, returning the winner. Four more tests park the put's insert behind a trigger and delete the winner before the put re-reads it, on hash and hash-range tables: the put retries and creates the item, and after 5 lost inserts it gives up with nothing written.Checklist
cargo test --workspace)cargo fmt --check)cargo clippy -- -W clippy::pedantic)Storagetrait, auth model, on-diskformat, or public CLI surface, an RFC has been accepted or is linked
below. Otherwise, an ADR captures the decision (link below).
ADR / RFC: n/a. One backend's create-race handling; no wire, trait, auth, on-disk, or CLI change.
Merge order: after the MongoDB fix for the same race, without which the new test gets HTTP 500s on MongoDB. The CI step conflicts by one line with #385: keep both
--testflags.Breaking changes
None.
By submitting this pull request, I confirm that my contribution is made under
the terms of the Apache License 2.0 and I agree to the Developer Certificate of
Origin (DCO). See CONTRIBUTING.md for details.