[vladwing]

Hourly semcode-index --lore refreshes grew one host's lore database from 3.9 GB to 20 GB in two days, and after about 14 runs the FTS indices stopped covering new mail, without a line in the log. These four commits stop the growth and make the failure visible; the prune they add assumes a single writer, explained below the list.

• Prune old versions of the lore tables after each lore run. Nothing pruned lore or lore_vectors, so every superseded FTS index stayed on disk (17 GB of the growth). The _indices count also kept rising until optimize_lore_fts_indices() refused to run. compact_lore_tables() now prunes both tables, and the _indices guard counts only non-empty entries.
• Merge partial batches before inserting them. At the end of its input every parser thread sends a short batch, and each became its own fragment. Inserters now combine short batches up to batch_size, so an hourly refresh adds one fragment to lore instead of one per parser.
• Buffer vectors before merging them into lore_vectors. The vector inserter ran one merge_insert per lore RecordBatch, each joining against the whole table: a day of new mail took about 1,600 of them and 56 minutes. It now writes once per 8,192 vectors.
• Report skipped FTS updates and compaction problems. The skip reasons went to tracing::warn!, which the default log level hides, and the success lines printed regardless. They now go to stderr, and the success lines print only when the work ran.

The prune passes delete_unverified, so it assumes no other process writes to the lore tables while a lore run is going: no overlapping lore runs and no separate --vectors run, as a systemd oneshot unit guarantees. optimize_database() still skips lore for the same reason.

This PR is independent of #112; they merge cleanly in either order.

Testing: on top of f6ba491, each commit passes cargo check --all-targets, and the branch passes cargo fmt --check, cargo clippy --all-targets -- -D warnings and cargo test (510 passed, 0 failed). The size and fragment numbers come from a host running hourly lore refreshes, and from a simulated refresh of 50 new emails.

[vladwing]

#113 needs one fix before it merges: on a production-sized (10-archive) lore DB, every lore run now prints a compaction warning for lore_indexed_commits. Which of the three fixes below would you take? Main leaves that table uncompacted as well, silently; #113 makes the skip visible. Otherwise #113 keeps the hourly refresh time flat and every row inside the FTS indices, and it slows lore fragment growth without bounding it. Full numbers for both PRs are in my comment on #112.

#113 alone. On a two-archive test DB (io-uring/0 + linux-nvme/0, with the last 6,000 commits of each archive held back and then published 125 per archive per run), the hourly --lore + --vectors pair stays at 1.5–1.8 s + 1.4–1.5 s across 6 runs, lore gains 2 fragments per run, and the FTS indices cover every row. On main the same pair takes 21.8 s + 26.5 s at run 1 (median of 3). Over a 24-run series, where the merged build stays at 1.4–1.8 s + 1.4–1.6 s, main reaches 288.9 s + 387.6 s and its FTS updates stop after run 16. Merged with #112, on the 10-archive DB (1,039,746 commits), a refresh takes 27–28 s against 13–14.5 minutes on main.

The warning. It reports a compaction skip, not a failed write: lore_indexed_commits passes the 500-fragment compaction limit during a cold build of the 10-archive DB and is never compacted after that. The cold build, the no-op run and both hourly runs that followed all printed:

Warning: Partial failure compacting table lore_indexed_commits (set SEMCODE_DEBUG=warn for details)

A cold build writes the table about once per 1,024-commit batch, which left 993 fragments on the merged build (3,160 on main). optimize_single_table() skips compaction above MAX_COMPACT_FRAGMENTS (500), and #113 now counts that skip as a partial failure. Each refresh then adds one fragment per archive (1,013 after two refreshes), and the skip message's advice, rebuild with --clear, recreates the same state. Main reached 3,660 fragments over the same series and printed nothing, so #113 alone would print the warning too; I ran #113 alone only on the two-archive DB. There the PR builds stay under 500, compact the table to one fragment every run and never warn. The table is about 29 MB, so the limit, which protects memory-heavy tables, guards nothing here. The fixes I see, none of which I have measured:

• Exempt lore_indexed_commits from the 500-fragment limit. Smallest change; compacting 1,000 fragments of SHAs should cost little memory.
• Compact lore_indexed_commits during the cold build, before it crosses 500. Keeps the limit and adds a compaction pass to long cold runs.
• Write lore_indexed_commits in batches larger than the 1,024-commit email batches. Removes the cause, but each inserter records a batch's SHAs right after storing that batch's emails; buffered SHAs are lost if a run dies, and the next run re-parses those emails only for the message_id check to skip them.

I have left out keeping the skip silent, as main does, because the table would then gain a fragment per archive on every refresh with nothing ever compacting it.

Also not fixed. lore still grows without bound, because it is never compacted, by design. On the two-archive DB the merged build adds 2 lore fragments per refresh (91 after the cold build, 145 after 24 hourly runs and a day-sized run), against about 240 per run on main. On the 10-archive DB it adds 8–9 per refresh (993 to 1,010 over two refreshes), against 114–140 on main: about 200 a day at hourly refreshes, and still unbounded. Pruning also leaves each superseded index's directory behind, about 5 per run: lore/_indices held 149 entries after the 26 runs that added mail, 15 of them non-empty. The FTS skip check stays clear only because #113 counts non-empty entries.

[vladwing]

I pushed an update that clears the permanent lore_indexed_commits compaction warning from my earlier comment with option (a), exempting that one table from the 500-fragment limit, plus two review fixes.

The new commit, "lore: compact lore_indexed_commits whatever its fragment count", adds the exemption in optimize_single_table() and keeps the limit for every other table. Databases already over the limit compact on their next lore run that adds mail.

I checked first that compacting a table of this shape is safe. On lance 4.0.0 a table with one SHA per row and no index compacts in under 5 s at both 1,013 and 3,660 fragments, with no thread spinning, whether alone or alongside five other tables compacting at the same time. The spin from 61e5493 does not reproduce on lance 1.0.1 either, the version semcode used at that commit, so its cause is still unknown, which is why the limit stays for the other tables.

The test builds 600 fragments in one write, which needs lance's WriteParams. lancedb does not re-export it, so the commit adds a lance dev-dependency pinned to the version lancedb already uses (4.0.0), and Cargo.lock gains no new packages.

The review fixes:

• create_lore_fts_indices() now uses the shared prune_old_versions() helper instead of its own copy.
• The lore_vectors progress bar advances on every batch it receives instead of only at each 8,192-vector write, so an incremental run no longer sits at zero until the end.

Still not fixed in this PR: lore itself still gains fragments on every refresh, and lance's prune leaves superseded index directories empty rather than removing them, which is harmless because the FTS guard counts only non-empty entries. A follow-up PR stacked on this one will keep the lore BTree indices current, and I am working on compacting the small lore fragments there too.