[vladwing]

Depends on #113: this branch is based on #113's head, so its first five commits are #113's. This PR adds the last two, each described below; review them one at a time.

lore: rebuild the non-nullable lore BTree indices each run

The lore BTree indices are created at schema setup, usually on an empty table, and nothing updates them afterwards: optimize_single_table() skips lore, and optimize_lore_fts_indices() merges only the FTS indices. In a database built from scratch they cover no rows, so every lookup by message_id scans the whole table, and so does every in_reply_to lookup a thread walk makes. A long-lived database had about 106,000 of its 998,000 emails outside them, a number every refresh adds to.

compact_lore_tables() now rebuilds, after pruning old versions, the BTree indices on non-nullable columns: message_id, git_commit_sha and date, plus from and subject until the first FTS build replaces those with FTS indices. It builds them from the whole table each run rather than merging only the new rows into them, so no earlier failed run, and no rewrite of the table's fragments, can leave rows out.

The BTrees on the nullable in_reply_to and references columns stay as they are, because lance 4.0.0 writes a BTree over a nullable string column of that size into pages it cannot decode: reading the index back panics, and so does any lookup that lands on a bad page. in_reply_to lookups therefore still scan the rows its index does not cover. A comment at the filter names the defect, so a lance upgrade can lift it.

A failed rebuild is reported as a partial failure, like a failed prune. The rebuild runs as its own task, so a panic inside lance is reported the same way instead of ending semcode-index before the FTS update. prune_lore_table() is renamed maintain_lore_table() to match.

On a copy of a production database (about 998,000 rows, load average 100 to 170 on 316 CPUs), rebuilding the three BTrees took 1.6 to 1.8 s every run, about twice as long as merging new rows, and peaked at 265 MB on its own. A run that rebuilds them peaks at 1.7 GB, most of it from listing the table's indices, which opens the FTS indices that the FTS update opens anyway. The index copies a rebuild replaces stay under _indices/ until the next run prunes them, adding one BTree set to the non-empty entries optimize_lore_fts_indices() counts against its limit of 100.

lore: compact the lore table once it has many small fragments

Every lore run appends a few fragments to lore, and nothing merges them. A production database had reached 970 fragments, most of them a few rows, and queries that scan lore pay for each one: on a compacted copy, the thread lookup's references scan used 27 to 40 s of CPU instead of 51 to 56 s, and a date-bounded subject search 1.0 to 1.3 s instead of 2.0 to 2.8 s. optimize_single_table() skips the table, because a compaction that remaps every index on it peaked at 8.9 GB on a production-sized table, and panics on the references BTree, which lance 4.0.0 writes in a form it cannot read back.

compact_lore_tables() now prunes, then compacts lore once it has more than 256 fragments, then rebuilds the BTrees as before. Fragments of 150,000 rows or more are not rewritten. Index remapping is deferred: lance records the old-to-new fragment mapping in its fragment-reuse index and leaves the indices as they are, which keeps memory low. The compaction runs as its own task, like the rebuild, so a panic in it is reported as a partial failure.

Reads and the FTS update apply the fragment-reuse mapping, but lance 4.0.0's BTree update does not: after such a compaction it drops every row the index held in a rewritten fragment, while the index statistics still report full coverage. That is why the BTrees are rebuilt from the whole table each run rather than updated, and a new test checks that lookups find every row after a compaction of already-indexed fragments.

#122, which encodes lore vectors in 64-email chunks, should land before or with this commit: larger fragments let update_lore_vectors() hand its workers full 1,024-row batches, which raises the peak of a cold --vectors run.

Costs and caveats:

• On a copy of the production database (998,000 rows, host load 130 to 200 on 316 CPUs), the first compaction took 970 fragments to 6 in 4.0 s, and the BTree rebuild after it 1.6 s, with a process peak of 3.3 GB.
• A table built from scratch by the lore: speed up indexing on many-CPU hosts #112/lore: keep hourly lore refreshes from growing the database without bound #113 lore pipeline holds only small fragments (997,703 rows in 1,043 fragments), so its first compaction rewrites all of it: 30 to 38 s and a 7.6 GB peak with the rewrite tasks run one at a time (8.4 GB at two, 9.0 GB at four), measured in a process of its own. About 3 GB of that is glibc arena fragmentation: with MALLOC_ARENA_MAX=2 it peaks at 4.2 GB.
• A cold build now ends with this compaction, in the process whose indexing peaked at 14.5 GB; that whole run's peak was not measured. Neither was an upstream-built table, which holds more, smaller fragments.
• Replaced fragments stay on disk until the next run's prune, so a query still reading them is not cut off. That is about 0.25 GB for an hour on production, and the whole table's data once on a cold database (2.3 GB more here).
• lance 4.0.0 never trims the fragment-reuse index, so each compaction adds a version to it, a few KB after the first. At 256 fragments a table compacts about every 25 hourly runs, so about 14 versions a year.