Repository navigation
Conversation
meta-cla
Bot
added
the
CLA Signed
|
Stress-test results for #112 and #113 together, against main: on a production-sized lore DB (10 archives, 1,039,746 commits), an hourly refresh ( Lore. Wall seconds, medians where n > 1. The two-archive DB is io-uring/0 + linux-nvme/0: 105,669 commits indexed cold, with the last 6,000 commits of each archive held back and then published 125 per archive per run (3,000 per archive for the day-sized run). The 10-archive DB holds 960,744 distinct message-ids after the cold build, with 25 commits per archive published per run.
After the 24 hourly runs the two-archive DB is 6.9 GB on main and 1.0 GB on the PR build. Main's FTS skip check (more than 100 Kernel (v7.3 tree, 64,869 files). Median wall seconds of 3 runs, with the range.
Max thread count drops from 961 to 677 on cold runs. Correctness. All 270 measured runs exited 0, and no PR-build run logged an error. Lore: after the cold and no-op runs (3 reps each) both builds hold the same row counts and the same number of distinct message_ids; after the day-sized run (3 reps) and at the end of the 10-archive series they also have the same order-independent digest over every non-vector column of Which PR does what. Each PR speeds up the cold lore build on its own, #112 cuts kernel system CPU, and #113 keeps hourly refreshes flat. Each branch ran alone on the two-archive DB (cold, no-op, 6 hourly runs, a day-sized run) plus one cold kernel run:
What was run. Both PRs at their current heads (be2d75f and 6b14580), merged locally, against main at f6ba491; main has since gained only #114, which touches |
insert_lore_indexed_commits() recorded each batch's SHAs with a merge_insert keyed on git_commit_sha. lore_indexed_commits has no index, so every merge_insert hash-joined the batch against the whole table. The join grew with every batch, and on a 316-CPU host most of the indexer's busy threads sat in that join while the email parsers waited on the inserters. The merge buys nothing here. The pipeline only hands it SHAs that were filtered against get_indexed_lore_commits() before parsing, and the readers treat the column as a set, so a rare duplicate row is harmless. Append with table.add() instead.
insert_lore_emails() filtered every batch through a `message_id IN (...)` query with one literal per email. lance plans that predicate once per fragment, and the BTree index on message_id only covers fragments that existed when it was last built, so each batch also scanned every fragment appended during the run. The cost of a batch grew with fragment count times batch size, and the indexer spent its time building and freeing DataFusion literal expressions. The check is still needed: a crossposted email has the same message_id in every list's archive, and a partially failed run can leave stored rows whose commit SHA was never recorded. So load the table's message_ids into a map with one unfiltered column scan (get_lore_message_ids()) when the first archive of a run with new mail needs them, and share the map between the inserters of every archive for the rest of the run. Once a pipeline's inserters finish, the map holds exactly the ids stored so far, this run's included, so a crosspost of an email stored from an earlier archive of the run is still caught. A run in which no archive has new commits never scans the column. LoreIdState, the map's value type, is exported next to DatabaseManager. Within an archive, a message_id can appear in more than one commit, and those copies can land in batches that different inserters store at the same time. Each batch claims its new ids as pending under the map's lock before appending, marks them stored afterwards, and releases the ids of emails that failed so another copy can still be stored. A copy whose id another batch has pending is deferred rather than skipped: it is reported with the failed emails, so its commit SHA is not recorded and the next run checks it against the table. Skipping it would record the SHA even if the other batch's insert then failed. The map costs one string per stored email. On a 316-CPU host, with the other two changes in this series applied, this took lore inserts from about 85 to 400-660 batches a minute.
#[tokio::main] starts one worker per CPU, and the lore pipeline started one email parser thread per CPU. On a 316-CPU host that is over 600 threads for a job limited by two database inserters: the parsers sat blocked on the result channel, and the process spent most of its CPU time in the kernel, in futex and spinlock contention from idle tokio workers parking and waking. Build the runtime by hand with min(num_cpus, 32) workers, still letting TOKIO_WORKER_THREADS override it as it does for the default runtime, and cap the lore parser threads the same way. The rayon pool is left alone; -j already sizes it. Machines with 32 CPUs or fewer see no change. The other modes do their CPU work on rayon, std threads or spawn_blocking, so the tokio cap does not touch them. The exception is lance's FTS index build, which tokenizes on tokio workers: a full lore FTS (re)build on a large host is now limited to 32 workers, and TOKIO_WORKER_THREADS lifts that. The database inserters are another matter. The inserter tasks of the --git, tree, --commits and lore pipelines are tokio tasks that block their worker in a std channel recv() while they wait for a batch, so with --db-threads at or above the worker count they can occupy every worker and starve the async work the inserters wait on. That was already reachable at one worker per CPU on small hosts; with the cap it is reachable at 32 inserters on any host. The defect predates this change, and facebookexperimental#119 moves those receives off the workers.
be2d75f to
0fad419
Compare
|
I pushed an update with two review fixes and one change to landing order: this PR should now land with or after #119. The reason is now in the tokio-cap commit's message. The database inserters of the Both review fixes are folded into "lore: check for duplicate message_ids in memory":
Four new tests cover the map:
|
On a 316-CPU host,
semcode-index --loreinserted about 85 batches a minute. With these three commits it inserts 400–660; the memory and thread-pool costs are listed after the commits. Each commit removes one bottleneck:insert_lore_indexed_commits()usedmerge_insert, which hash-joined every batch against the wholelore_indexed_commitstable. That table has no index, so the join grew with every batch. The SHAs are already filtered againstget_indexed_lore_commits()before parsing, so a plaintable.add()is enough.message_id IN (...)query was planned once per fragment and scanned every fragment appended since the last index build. The message_ids are now loaded once per archive and shared by that archive's inserters, which claim ids under a lock so concurrent batches cannot store the same email twice.TOKIO_WORKER_THREADSstill overrides the cap. Hosts with 32 CPUs or fewer see no change.The speedup has three costs:
TOKIO_WORKER_THREADSis raised.--db-threadsof 32 or more could now starve the runtime. The default of 2 is unaffected.Testing: on top of f6ba491, each commit passes
cargo check --all-targets, and the branch passescargo fmt --check,cargo clippy --all-targets -- -D warningsandcargo test(510 passed, 0 failed).