[dtor]

Summary

lore.kernel.org uses public-inbox v2 repositories partitioned into numbered epoch shards (git/0.git, git/1.git, ..., git/N.git). When a shard reaches its size limit (~1 GB), lore.kernel.org permanently seals that shard and creates <N+1>.git (for example, lkml/git/20.git superseding lkml/git/19.git).

Previously, semcode-index --lore (with no arguments) only ran git fetch on existing <db_path>/lore/<list_name>/<N> directories already present on disk. As a result:

1. Once a mailing list rolled over to <N+1>.git, semcode-index --lore never discovered or cloned the new shard and silently stopped receiving new emails for that list.
2. Sealed older <list>/<M> bare git repositories remained on disk indefinitely, even though all of their emails were already stored in the LanceDB lore table and git fetch on them was a perpetual no-op.
3. On high-core-count workstations, bulk-indexing a full shard (~240k emails) experienced severe memory bloat (~103 GB RSS) and O(B²) database scan overhead because every batch of 500 emails executed a full scan of lore.message_id via try_collect(), while num_cpus::get() worker threads (e.g., 128+ threads) simultaneously buffered num_workers * batch_size full email bodies in RAM.

This series fixes shard rollover discovery, retires superseded local git shards once verified complete, and reduces peak bulk-indexing memory on high-core-count workstations from ~103 GB to ~3.7 GB.

---

Changes

1. semcode-index: automatically sync new lore shards and retire old ones

• Shared Manifest Discovery (list_manifest_lore_archives, find_new_lore_archives, sync_new_lore_archives):
  • Factor out list_manifest_lore_archives() to parse /<list_name>/git/<N>.git entries from https://lore.kernel.org/manifest.js.gz and share it between resolve_lore_url() and find_new_lore_archives().
  • During semcode-index --lore (both standalone and combined refresh), check manifest.js.gz for any shards <N> strictly greater than the highest local shard <max_local> for each tracked mailing list, and automatically clone and index them.
• Causal Ordering & Safe Retirement (retire_superseded_lore_archives):
  • Run sync_new_lore_archives() before fetching and indexing existing local shards. Because public-inbox v2 permanently seals shard M before publishing shard N > M in manifest.js.gz, observing N in manifest.js.gz before running git fetch on M guarantees that M's fetch retrieves all of its final tail commits.
  • Once all shards for a mailing list in the current run report fully_indexed == true (recorded_commits == new_count) with no fetch or indexing errors, remove the local bare git directories for superseded shards (< max_num) while keeping <max_num> on disk as the high-water mark. All indexed emails remain permanently stored in the LanceDB lore table.
• public-inbox Deletion Commits (LoreCommitParseOutcome::NoEmailFile):
  • Handle public-inbox deletion (rm) commits (whose tree contains 'd' instead of 'm') by recording their commit SHAs in lore_indexed_commits without deleting existing rows from lore, preventing them from remaining permanently unindexed and blocking shard retirement.
  • Resolve each commit and its 'm' blob in a single pass (parse_email_from_blob_content) and filter failed insertions by message_id so duplicate-message_id commits in a batch are never prematurely recorded in lore_indexed_commits on error.

2. database: cache lore message_ids in memory during bulk insertion

• Cache known message_ids in DatabaseManager::lore_message_id_cache (tokio::sync::Mutex<Option<HashSet<String>>>):
  • On cold start, stream only the message_id column from the lore table one RecordBatch at a time via try_next() (dropping each Arrow buffer immediately instead of materializing all batches via try_collect()).
  • Atomically claim candidate message_ids in the cached set under the mutex so concurrent inserter tasks (db_threads > 1) cannot insert duplicate message_ids across concurrent batches, and roll back any failed_indices on insertion error.
  • Stream get_indexed_lore_commits() via try_next() as well, and invalidate lore_message_id_cache in clear_all_data(), compact_lore_tables(), drop_and_recreate_tables(), and drop_and_recreate_table("lore").

3. indexer: bound in-flight memory and worker count in lore pipeline

• Cap git/email parsing worker threads in process_lore_commits_pipeline() to num_workers.clamp(1, 16) so high-core-count machines do not spawn dozens of idle git pack-cache and parsing threads when bottlenecked on LanceDB writes.
• Replace per-worker Vec<LoreEmailInfo> batch buffers (num_workers * batch_size emails held concurrently across worker threads) with a bounded mpsc::sync_channel(batch_size * 2) feeding a single batch collector thread and a bounded tokio::sync::mpsc::channel(4) feeding the async database inserters.
• Join OS worker/collector/producer threads inside tokio::task::spawn_blocking so Tokio executor threads remain free to drive the async inserter tasks.

---

Testing & Verification

• Verified full indexing of lkml/19 (222,448 commits, including public-inbox deletion commits) followed by semcode-index --lore automatic discovery of lkml/20, tail catch-up, and automatic removal of the superseded lkml/19 git clone directory.
• On a high-core-count workstation, peak RSS during full shard indexing dropped from 103.3 GB to ~3.75 GB, full initial shard indexing completed in ~60s (down from 51m57s CPU time), and incremental lkml/19 -> lkml/20 rollover completed in ~20s.
• Verified FTS and message-id thread queries (lore_search, dig) across emails from both lkml/19 and lkml/20.
  dtor@dtor-ws:~/kernel/linux-next (b4/geode-swnode)$