Skip to content

symbol_filename: serialize upserts - #116

Open
vladwing wants to merge 1 commit into
facebookexperimental:mainfrom
vladwing:symbol-filename-upsert-lock
Open

vladwing wants to merge 1 commit into
facebookexperimental:mainfrom
vladwing:symbol-filename-upsert-lock

Conversation

@vladwing

Copy link
Copy Markdown
Contributor

Concurrent upserts can store one (symbol, filename) pair twice. Six identical cold runs of a build of main ended with symbol_filename row counts up to 99 apart.

insert_batch_and_mark() runs a batch's functions and types inserts with tokio::join!, and both end in SymbolFilenameStore::insert_batch(). Several inserters also run at once. When two merge_inserts of the same new pair open the table at the same version, both miss and both insert.

Serialize insert_batch() with a process-global async mutex, held across open_table and every chunk's merge_insert: the pattern content.rs uses for its shards. Every symbol_filename upsert in a process now waits for the one before it; the cost to a cold index is not measured. The mutex is a static rather than a field so that stores of different DatabaseManagers opened on one database also wait on each other.

The new test stores every pair twice without the lock (5,120 rows for 2,560 pairs) and once with it. It runs two insert_batch() calls concurrently with the same 64 new pairs, for 40 rounds with fresh pairs each round.

Concurrent upserts can store one (symbol, filename) pair twice. Six
identical cold runs of a build of `main` ended with `symbol_filename`
row counts up to 99 apart.

`insert_batch_and_mark()` runs a batch's functions and types inserts
with `tokio::join!`, and both end in
`SymbolFilenameStore::insert_batch()`. Several inserters also run at
once. When two `merge_insert`s of the same new pair open the table at
the same version, both miss and both insert.

Serialize `insert_batch()` with a process-global async mutex, held
across `open_table` and every chunk's `merge_insert`: the pattern
`content.rs` uses for its shards. Every `symbol_filename` upsert in a
process now waits for the one before it; the cost to a cold index is not
measured. The mutex is a static rather than a field so that stores of
different `DatabaseManager`s opened on one database also wait on each
other.

The new test stores every pair twice without the lock (5,120 rows for
2,560 pairs) and once with it. It runs two `insert_batch()` calls
concurrently with the same 64 new pairs, for 40 rounds with fresh pairs
each round.
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Nov 11, 2026
@vladwing vladwing closed this Nov 11, 2026
@vladwing vladwing reopened this Nov 11, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant