feat(index): full-text search with BM25 ranking (#70)

[CaffeinatedCoder]

Part of #70.

Draft, stacked on #169, #170, #172, #173, #174 and #175. The branch merges those fix branches, so until they land this PR shows their commits too. Only the full-text commits belong here: feat(index): full-text search with BM25 ranking (0d062b1) and the review follow-ups fix(fts): address review of full-text search (dbedc42), 5c68c46, c2616e2, f4600e3 and c663036. It relies on the fixes: index entries removed on update/delete (#172), indexes filled from existing documents (#175), and rebuilds into fresh trees that don't revive stale pages (#169, #170). I'll rebase once they are merged.

What it does

A full-text index type for string properties, built the way #70's comment suggested: a companion index per collection, not a virtual table.

// Declare in the model, or create at runtime; filled from existing documents either way.
modelBuilder.Entity<Article>().HasFullTextIndex(a => a.Body);          // idx_fts_Body
await db.Articles.CreateFullTextIndexAsync(a => a.Title);              // idx_fts_Title

// LINQ: documents containing every term, served from the index.
var hits = db.Articles.AsQueryable()
    .Where(a => a.Body.FullTextMatch("quick fox") && a.Year > 2000)
    .ToList();

// Ranked: top k by BM25, with scores.
await foreach (var r in db.Articles.FullTextSearchAsync("idx_fts_Body", "quick fox", k: 10))
    Console.WriteLine($"{r.Score:F3}  {r.Document.Title}");

// matchAllTerms: false → any term qualifies, ranked by relevance.
db.Articles.FullTextSearchAsync("idx_fts_Body", "apple durian", 10, matchAllTerms: false);

// Schemaless
await col.CreateFullTextIndexAsync("text");
col.FullTextSearchAsync("idx_fts_text", "quick fox", 10);

Design

Analysis. FullTextAnalyzer reuses TextNormalizer.Normalize: accents removed, lowercased, anything that isn't a letter or digit treated as a separator. It then splits into terms, cutting each term at 32 bytes of UTF-8 to keep keys small. Documents and queries are analyzed the same way.

Storage. One B-Tree per index (IndexType.FullText = 6), with two kinds of keys:

Key	Points at
[0x10][term][0x00][location (6)][term frequency u16][document length u32]	the document
[0x20][documents u64][total terms u64] (single entry)	—

• Keys use the document's location rather than its ID, so their size doesn't depend on the ID type (string IDs are unbounded).
• Term frequency and length in the key mean BM25 needs nothing beyond the postings.
• A delete re-analyzes the old text and removes exactly those keys. Updates and deletes are therefore precise and transactional, and a rollback restores the index.
• The statistics entry is updated in the same transaction as the postings.
• The tags sit outside the [0x00..0x03] keys an older BLite writes when it reads the tree as a plain B-Tree index.
• FullTextIndex.FormatVersion (analyzer and key layout) is stored in the existing IndexMetadata.KeyFormat field. On open, an index with another version, or holding keys it did not write, is rebuilt into a fresh tree (typed and dynamic). Otherwise a full-text index persists like a B-Tree index with type 6.
• Every hit is re-checked against the document text, so a stale posting can never surface another document.
• A typed rebuild on open persists the emptied tree with key format 0 and records the current format only after the rebuild commits, so a failed or interrupted rebuild is retried on the next open.
• FullTextSearchAsync keeps taking candidates past the first k until k hits survive the re-check. Postings are read once per location.
• Two explicit write transactions on one collection lose one statistics update (single stats entry, no write-write conflict detection). Results stay correct; scores drift until a rebuild. Search falls back to estimates from the postings if the entry is missing.

Matching.

• LINQ FullTextMatch requires every query term, like SQLite FTS5's implicit AND. It has a real in-memory implementation with the same rules, so it is also correct inside ||, under !, or on a property without an index (the lesson from Near / Within always return true outside the index: non-indexed spatial queries match every document, and indexed Near returns the whole bounding box #165). The index-served plan is Exact; with && the other conditions become a post-filter.
• FullTextSearchAsync scores with BM25 (k1 = 1.2, b = 0.75, idf = ln(1 + (N − df + 0.5)/(df + 0.5))).

Planner. Equality, range and IN lookups now skip vector, spatial and full-text indexes. Before, any index covering the property qualified, so x.Title == "..." with a full-text index on Title would have been planned as a B-Tree lookup on it. A vector or spatial index on the property already had the same problem.

Index creation. It goes through #175's build-then-register path. CreateFullTextIndexAsync also holds the collection write lock while filling, so a concurrent insert cannot miss the index.

Also wired up: freeing pages when the collection is dropped, VacuumAsync(RebuildIndexes) (typed and dynamic), BLiteMigration, and DynamicIndexDescriptor.IsFullText.

Tests

FullTextSearchTests, 33 tests:

• Matching: in-memory accents, case and punctuation; LINQ AND semantics; &&, || and ! combinations; Count and First.
• Planner: FullTextMatch uses the full-text index; equality on a property with both index kinds uses the B-Tree; && post-filters; no plan without an index.
• Ranking: BM25 ordering (term frequency and length normalization, rare beats common), k, matchAllTerms: false.
• Statistics: scores after updates and deletes equal the scores of a freshly built index over the same documents.
• Maintenance: in-place and relocating updates, deletes followed by slot reuse, rolled-back writes.
• Lifecycle: reopen keeps identical scores; an index added to existing data (model and runtime, idempotent); a vacuum rebuild keeps results and scores; a dropped index falls back to scanning.
• Dynamic API: create on existing data, update, delete, reopen.
• On-disk robustness: keys written by an older version (ignored, then rebuilt on open); format-version mismatch rebuild (typed with scores equal to a fresh index, dynamic for both triggers); no rebuild of current indexes; stale postings never surface a document (typed and dynamic); missing statistics; name collision with a B-Tree index; terms over 32 bytes cut mid-character; planner skips FullTextMatch under || and !; a failed rebuild triggered by foreign keys is retried on the next open; stale hits in the top k are replaced (typed and dynamic); a stale posting that shares its location with a current one is counted once.

Full suite on macOS: 2479 passed, 0 failed, 4 skipped.

Performance (rough, macOS M-series, Release, 5,000 documents of 50 words from a 5,000-word vocabulary)

	without index	with index
InsertBulkAsync (5,000)	86 ms	901 ms
200 single InsertAsync	11 ms	1,523 ms
200 single UpdateAsync	17 ms	3,368 ms
200 updates in one transaction	4 ms	374 ms
ranked search (2 terms, k = 10)	—	~0.3 ms
LINQ Count (2-term AND)	—	43 ms

Searches are cheap. Search(matchAllTerms: true) intersects rarest-first before scoring: once the candidates are few, the postings of a common term are looked up per candidate instead of being decoded. The saving is decoding and allocation, not I/O, because counting each term (CountRange) still walks all of its leaf entries. Single-document writes are expensive: a document's ~40 distinct terms land on ~40 different leaf pages, and the WAL logs whole pages, so each autocommit writes ~40 × 16 KB. Batched writes (InsertBulkAsync, explicit transactions) share pages and stay fast. Documented on HasFullTextIndex/CreateFullTextIndexAsync. A segment-based store is a separate design.

Not in this PR (follow-ups)

• Highlighting: FullTextHighlighter.Highlight(text, query, "<b>", "</b>") as requested in Full-text Search (FTS) #70. It is stateless and small; next PR.
• Prefix queries (fox*), a cheap range scan over [0x10][prefix].
• More query and index features: BLQL $text, Studio index UI, analyzer options (stop words, minimum length, stemming), multi-field indexes.
• Ordering by relevance in LINQ. Use FullTextSearchAsync for ranking.

Compatibility / release notes

Do not open a database with full-text indexes for writing with BLite < 5.3.

An older version reads a full-text index as a B-Tree index. Its inserts and updates leave keys behind that 5.3 detects and repairs with a rebuild on open. Its deletes can leave stale postings behind, which the hit re-check hides. The page format version is unchanged, so older versions can still open the database. A per-collection "required features" value that lets future versions refuse such a collection is tracked in #187 and will land before any release that ships full-text search.

API additions: IDocumentCollection<TId, T> gains CreateFullTextIndexAsync and FullTextSearchAsync. Third-party implementations of the interface must add them, as with UpsertAsync. DynamicCollection gains CreateFullTextIndexAsync(fieldPath, name) and FullTextSearchAsync, and DynamicIndexDescriptor gains IsFullText.

Analysis caveat: scripts without spaces between words (Chinese, Japanese) form one term per run of letters, cut to about ten ideographs. Characters outside the Basic Multilingual Plane are separators.

🤖 Generated with Claude Code

[mrdevrobot]

Good draft, thanks for the thorough description. Analyzer, key encoding (0x00/0x01 can't occur in a term, truncation backs off to a code-point boundary, the [0x01][term][0x00]..[0x01] range is exact), BM25 (Lucene idf form), planner fallbacks for || and !, the SupportsKeyLookup change (a fix, not a regression), locking in CreateFullTextIndexAsync, and netstandard2.1 all check out. 19/19 new tests pass, full suite green apart from the known .wal-shm failures. Two on-disk points should change before this leaves draft, since they can't change after release.

1. The stats tag 0x02 is IndexKey.KeyPrefix, the prefix of every legacy B-Tree key (FullTextIndex.cs:24-26, :147-170; IndexKey.cs:55-60). An older BLite loads a type-6 index as a plain BTreeIndex and its InsertKey writes [0x02][encoded string] into the FTS tree. After re-upgrading, ReadStats/AdjustStats scan [0x02]..[0x03] and take the first junk key as the stats entry: garbage counts, or ArgumentOutOfRangeException from key.Slice(9) on keys shorter than 17 bytes, on every search and insert. Use tags that can't appear in legacy keys (e.g. 0x10 posting, 0x20 stats), validate key.Length == 17 in ReadStats/AdjustStats, and treat anything else as "rebuild needed".

2. No per-index analyzer/format version (FullTextIndex.cs:56-70). Delete relies on the current analyzer reproducing the stored keys, and FullTextAnalyzer delegates to the public, embedding-oriented TextNormalizer.Normalize. Any change there leaves orphan postings pointing at locations later reused by other documents, and since the LINQ plan is Exact with no post-filter (DocumentCollection.cs:3591-3593) those become false positives. Reuse IndexMetadata.KeyFormat/NeedsKeyFormatMigration (currently gated on decimal B-Trees, CollectionSecondaryIndex.cs:631-642) with an FTS format byte and rebuild on open when it differs. Consider also post-filtering FTS hits with FullTextAnalyzer.Matches, it's cheap next to loading the document. This also unlocks CJK bigrams later without a manual rebuild.

3. The single stats key makes concurrent explicit transactions a guaranteed lost update (FullTextIndex.cs:152-170). _collectionLock is per operation, not per transaction, and there is no write-write page conflict detection, so two explicit transactions inserting into the same FTS collection both rewrite the stats key on private copies of the same leaf; last commit wins. Either keep the document count out of the tree (the collection already tracks it) and store only total length, or state that concurrent write transactions on one collection aren't supported. Related: AdjustStats drops the entry at count 0 (:165), after which Search returns nothing while Match still works.

4. CreateFullTextIndexAsync silently returns a non-FTS index of the same name (DocumentCollection.cs:1010-1019), and FullTextSearchAsync then throws later; the dynamic variant throws "already exists". Check index.Type and throw on mismatch.

On your questions:

1. Write cost: not a blocker for v1. The ~40 dirty leaves per single-document autocommit are inherent to a term-major layout; a posting buffer doesn't reduce the leaves touched by a single-document transaction, and for batches the transaction-private page cache already coalesces writes. Document "batch inserts in one transaction" on HasFullTextIndex/CreateFullTextIndexAsync and leave a segment-based store as a separate design. Cheap win now: Search(matchAllTerms: true) reads every posting of every term; intersect rarest-first like Match does before scoring.
2. Downgrades: nothing shipped today validates a format version on open (PageHeader.FormatVersion is written, never checked), so no bump can make older versions refuse. Fix #1 so legacy keys are detectable, add the FTS format byte from #2 so the new version self-heals, and start recording a per-collection "required features" value now so future versions can refuse. Don't bump the page format version, that would lock every old version out of every database. Release notes: "do not open a database with full-text indexes with BLite < 5.3 for writing".
3. CJK: fine as a documented limitation. Note that char.IsLetter is true for Han/Kana/Hangul, so a sentence becomes one term truncated to 32 bytes (~10 ideographs), and supplementary-plane letters/emoji are dropped. Bigrams for CJK runs are a natural follow-up once #2 is in.

Nits: IDocumentCollection<TId,T> declares CreateVectorIndexAsync but not CreateFullTextIndexAsync; DynamicCollection.FullTextSearchAsync has no CancellationToken; CreateFullTextIndexAsync doesn't pass ct to the rebuild; missing tests for a >32-byte multi-byte term, the ||/! fallbacks, and the legacy-key collision once #1 is fixed.

[CaffeinatedCoder]

Thanks for the careful review. Addressed in dbedc42 (and docs in 5c68c46). The branch now also merges the latest review commits of the stacked fix branches.

1. Stats tag collides with legacy keys. Postings are tagged 0x10 and the statistics entry 0x20, so the generic [0x00..0x03] keys an older BLite writes fall outside every range the index reads. ReadStats/AdjustStats skip any entry that isn't 17 bytes.

I went one step further on "treat anything else as rebuild needed". 5.2.x loads a type-6 index as a B-Tree and writes its KeyFormat byte back unchanged, so the format byte from #2 can't tell that 5.2 wrote to the index. Its leftover keys can: FullTextIndex.HasForeignKeys() checks the first key and the keys after the postings (two range starts per index). If it finds anything it didn't write, the index is rebuilt on open, for both typed and dynamic collections. An old version's inserts and updates leave such keys behind. Its deletes leave stale postings instead, which the hit re-check below covers.

2. Analyzer/format version. FullTextIndex.FormatVersion (1) is recorded as the index's IndexMetadata.KeyFormat.

• Typed: NeedsKeyFormatMigration now also covers full-text indexes, so the existing migration rebuilds them into a fresh tree on open and records TargetKeyFormat.
• Dynamic: a stale index is rebuilt into a fresh tree, swapped in after the commit, persisted, and the old pages freed.
• Re-check: every hit is now checked against the document text. The LINQ executor applies the predicate to full-text hits even when the plan is Exact, and FullTextSearchAsync (typed and dynamic) checks Matches/MatchesAny. A stale posting can no longer surface the document that now holds its location.

3. Single stats key and concurrent transactions. I documented it on FullTextIndex: two explicit write transactions on one collection lose one statistics update. Matching doesn't depend on the statistics, so results stay correct; only the scores drift until a rebuild. Keeping the document count out of the tree would only halve the problem, because the total length has the same lost update. A real fix needs write-write conflict detection or statistics derived from a segment store.

On the count-0 case: when the entry is missing, Search now estimates N and the average length from the postings it already read, instead of returning nothing.

4. CreateFullTextIndexAsync over a non-FTS name now throws InvalidOperationException naming the index.

On your answers:

1. Write cost: batching advice is documented on HasFullTextIndex and CreateFullTextIndexAsync. Search(matchAllTerms: true) now intersects rarest-first before scoring. Once the candidates are 8× fewer than a term's postings, it looks each candidate up instead of reading that term's full posting list.
2. Downgrades: done as you suggested; the page format version is untouched. I added the release-note line to the PR description. I'd like to do the per-collection "required features" value as its own small PR, since it changes the metadata format for every collection rather than just full-text indexes. Is that OK, or do you want it in this one?
3. CJK: documented on FullTextAnalyzer: a run of ideographs is one term cut to about ten characters, and supplementary-plane characters are separators. Bigrams can come later behind a FormatVersion bump, which now rebuilds on its own.

Nits: IDocumentCollection<TId,T> declares CreateFullTextIndexAsync, DynamicCollection.FullTextSearchAsync takes a CancellationToken, and the typed rebuild gets ct.

Tests (11 new, 30 total). Each one fails with its fix reverted.

• Legacy keys: generic keys don't change results or scores before the rebuild, and are gone after reopen.
• Format mismatch: a stale typed index is rebuilt, removing a stale posting, with scores equal to a freshly built index. A dynamic theory covers both triggers (format byte and foreign keys) and checks the index isn't rebuilt again on the next open.
• No false rebuilds: indexes written by this version, including an empty tree, keep their root on reopen.
• Stale postings: never surface the document at their location (LINQ list, Count, FirstOrDefault, and ranked search with all or any terms), for typed and dynamic collections.
• Missing statistics entry: search still finds and scores the documents.
• Name collision: creating a full-text index under a B-Tree index's name throws.
• Terms over 32 bytes: 2- and 3-byte characters where the cut falls mid-character. Matching uses the cut term, and an update removes the cut keys from the index itself.
• Planner: FullTextMatch under ||, || FullTextMatch and ! is not served by the index. Execution of those cases was already covered by Linq_FullTextMatch_CombinesWithOtherConditions.

Full suite on macOS: 2476 passed, 0 failed, 4 skipped. BLite.NetStandard21.Tests builds, but I couldn't run it here (no .NET 6 runtime installed).

🤖 Generated with Claude Code

[mrdevrobot]

All four points are fixed, and I checked that the legacy-key, format-mismatch and stale-posting tests fail with their fixes reverted. HasForeignKeys covers every first byte an older version could write, and on open it costs two short range reads. The re-check doesn't touch BM25, and Count() doesn't get slower. Full suite 2476 / 0 / 4 skipped, netstandard2.1 builds with no warnings. One defect left.

A typed migration triggered only by foreign keys is never retried after a failure. MigrateLegacyDecimalIndexes → RecreateIndex (CollectionIndexManager.cs:286-297) persists the empty tree and frees the old one before the rebuild, and copies KeyFormat = old.KeyFormat. When the trigger was HasForeignKeys(), that format is already 1. If the rebuild throws (an unreadable document) or the process crashes, the empty tree has no foreign keys and the right format, so the next open trusts it and the index stays silently empty. The message "the migration is retried on the next open" is wrong for this case. Fix: persist KeyFormat = 0 for a full-text index before the rebuild, or build unregistered and swap after commit as the dynamic path does. Please add a test that fails the rebuild and reopens.

On "required features": a separate PR, merged before any release that ships FTS. It is a general mechanism and needs its own compatibility tests. It also covers cases this PR can't detect:

• An older DynamicCollection doesn't load type-6 indexes, so it never writes postings and leaves no foreign keys. Its PersistIndexMetadata also drops the index from the metadata and leaks its pages.
• An older typed version that only deletes leaves stale postings. The re-check hides them, but they inflate document frequency.

Minor, only reachable in stale states: FullTextSearchAsync re-checks after taking the top k, so it can return fewer than k; and Match can return a location twice if a stale posting shares it with a new document. On the docs: CountRange in the rarest-first step is still an O(df) leaf scan, so the saving is decoding and allocation, not I/O.

The IDocumentCollection additions are fine; there's precedent with UpsertAsync. Please mention them in the release notes.

[CaffeinatedCoder]

Thanks. Fixed in c2616e2, f4600e3 and c663036.

Retry after a failed typed migration (c2616e2): RecreateIndex now persists the emptied tree as LegacyKeyFormat (0) instead of copying the old format. The target format is written by SetKeyFormat only after the rebuild commits. A full-text index treats any format other than FullTextIndex.FormatVersion as needing migration, so the next open rebuilds again whatever the trigger was. Decimal B-trees are unaffected (their old format was already 0), and the "retried on the next open" message is now accurate.

FailedRebuildTriggeredByForeignKeys_IsRetriedOnTheNextOpen:

• adds a foreign key while the format stays current, then corrupts one document;
• asserts that the open throws and the stored format still asks for migration;
• repairs the page, reopens, and checks the index is complete.

With the fix reverted it fails. Without the metadata assertion it still fails, because the reopened index is empty.

Required features: agreed, a separate PR merged before any release that ships FTS. Filed as #187 with both cases you listed.

Minor (f4600e3):

• FullTextSearchAsync (typed and dynamic) now takes candidates in widening windows (k, 2k, …, skipping hits already returned) until k survive the re-check. The first window is a single Search, as before.
• Postings yields one posting per location. A stale posting with the same term and location sorts next to the current one, so Match no longer returns the location twice and Search no longer scores it twice.
• Tests: StaleHitsInTheTopK_AreReplacedByTheNextCandidates, StalePostingAtTheLocationOfADocumentWithTheSameTerm_IsCountedOnce, and a k = 1 case in the dynamic stale-postings test. All fail with the fix reverted.

Docs (c663036): the comments and the PR description now say that rarest-first saves decoding and allocation, not I/O, since CountRange reads every leaf entry of each term.

IDocumentCollection: the new members are listed in the release notes section of the description.

Full suite: 2479 passed / 0 failed / 4 skipped. netstandard2.1 builds with no warnings.

[mrdevrobot]

Verified c2616e2, f4600e3 and c663036.

• Retry: with an injected foreign key and one corrupted document, three opens in a row each throw and the stored KeyFormat stays 0; after repairing the page the next open fills the index, and a further open doesn't rebuild. Decimal and non-decimal B-Trees never reach RecreateIndex with a non-zero format, so nothing re-migrates.
• k results: with stale postings outranking every real hit plus a stale duplicate, k = 1..5 returns min(k, 3) results, no duplicates, no stale documents.
• Both fail with the fixes reverted. 33/33 full-text tests, full suite 2479 / 0 / 4 skipped.

Minor, not blocking: Postings (FullTextIndex.cs:275-284) keeps the first posting for a location in key order, which sorts by frequency then length, so a stale posting can be the one used for scoring. In a probe a stale length-1 posting at a document's location moved its BM25 score from 1.0203 to 1.3075, and CountRange counts both for document frequency. Membership and order stay right. A comment is enough; StalePostingAtTheLocationOfADocumentWithTheSameTerm_IsCountedOnce compares titles only.

Rebase needed. #169 and #170 are now on main as squashes, so this branch conflicts. Please rebase onto main and drop the pre-squash #167/#168 commits and merges. Most conflicts resolve to main's side (StorageEngine.Recovery.cs, FreePageReuseTests.cs, VacuumTests.cs, the vacuum failure path in DocumentCollection.cs). Two need your side:

• CollectionSecondaryIndex.cs, CollectAllPages: keep if (_fullTextIndex != null) return _fullTextIndex.CollectAllPages(...). Without it a failed full-text vacuum or swap leaks the copy's pages.
• CollectionIndexManager.cs, RecreateIndex: keep LegacyKeyFormat; main's fresh.KeyFormat = old.KeyFormat would undo the retry fix. Keep main's ReplaceIndex, which restores the old index when SaveMetadata fails.

I did a trial resolution this way and the full suite passed (2483 / 0 / 4 skipped). #172, #173, #174, #180 and #182 are landing now, so the rebase will also pick them up.

[CaffeinatedCoder]

Rebased onto main (efacaf4); the branch is now linear with no merge commits.

Commits

• Dropped the pre-squash Spatial and vector indexes are never updated on UpdateAsync / DeleteAsync: stale and unrelated documents come back #164/Near / Within always return true outside the index: non-indexed spatial queries match every document, and indexed Near returns the whole bounding box #165/VacuumAsync(RebuildIndexes = true) duplicates every secondary-index entry on typed collections #167/A freed page that gets reused reads back its old contents from the WAL: dropping and recreating an index duplicates entries #168 commits and every merge.
• Dropped Near misses points across the ±180° meridian and over the poles #166 (3450e15, 8ba4628): it only touches RTreeIndex/SpatialMath, and full-text search doesn't use either.
• Kept An index declared with HasIndex on a collection that already has documents stays empty: queries through it miss every existing document #171 (f00a13b, 9c113ed, 89578bb): full-text search builds on it. CreateFullTextIndexAsync goes through An index declared with HasIndex on a collection that already has documents stays empty: queries through it miss every existing document #171's build-unregistered/fill/register path in ApplyIndexBuilder, and ResolveIndexName, which An index declared with HasIndex on a collection that already has documents stays empty: queries through it miss every existing document #171 adds, gains the index-type parameter here. These are fix(index): fill model-declared indexes from existing documents (#171) #175's commits up to 1cb3a8e. fix(index): fill model-declared indexes from existing documents (#171) #175 still gets the changes from your latest review; once it lands, these three drop out on the next rebase.
• Full-text commits: cecde76, de65273, d731941, f8c6183, 79b11a1, cc049d4, plus 17832e3 (below).

Resolution

• CollectionSecondaryIndex.CollectAllPages(transactionId): main's version plus if (_fullTextIndex != null) return _fullTextIndex.CollectAllPages(transactionId);. The FullTextIndex.CollectAllPages(ulong) overload, which used to come in through a merge, is now part of cecde76.
• RecreateIndex keeps LegacyKeyFormat, together with main's ReplaceIndex, which restores the old index when SaveMetadata fails.
• IndexOptimizer: main's lines, plus the SupportsKeyLookup guard on the Contains/IN path.
• Everything else is main's side. With cherry-picks, the Recovery/FreePageReuse/Vacuum conflicts don't come up.

Stale posting scoring (17832e3): the Postings doc comment now says it returns the first posting of a location in key order (lowest frequency, then shortest length). That posting can be the stale one used for scoring, and CountRange counts both towards document frequency. Membership and once-per-location are unaffected. The test now notes that it checks membership and order only.

Tests: full suite 2495 / 0 / 4 skipped; full-text 33/33.

#194 merges cleanly on top of this (2506 / 0 / 4 in a trial merge). Once both are in, full-text indexes will record a required feature as a follow-up.

[CaffeinatedCoder]

Rebased onto #175's current head (758717b). This PR is now stacked on #175 and should be merged after it; the #171 commits are no longer part of this branch.

• MigrateLegacyDecimalIndexes uses fix(index): fill model-declared indexes from existing documents (#171) #175's copy-and-swap: CreateUnregistered, then FillUnregisteredIndexAsync, then PublishFilledIndex/ReplaceIndex. The copy gets the index's own target format (TargetKeyFormat, which is FullTextIndex.FormatVersion for full-text). RecreateIndex stays removed, and the LegacyKeyFormat workaround from c2616e2 goes with it. A failed rebuild now leaves the old index registered as it was, foreign keys included, so NeedsKeyFormatMigration still holds and the next open retries.
• CreateFullTextIndexAsync fills through FillUnregisteredIndexAsync(index, ct), so a failed fill frees all its pages. CollectAllPages keeps the _fullTextIndex branch.
• 8ab0ad3 reworks FailedRebuildTriggeredByForeignKeys_IsRetriedOnTheNextOpen. Over three failing opens, the stored root and format stay unchanged, the foreign key is still there and the file doesn't grow. After the repair, the next open swaps in the rebuilt index, and a further open keeps it. The test fails if a failed fill publishes an empty copy, if the copy's pages aren't freed, or without the _fullTextIndex branch in CollectAllPages.
• 50df46c documents the stale-posting scoring caveat on Postings.

Full suite 2499 / 0 / 4 skipped; full-text 33/33, DecimalIndexTests 11/11, IndexBackfillTests 8/8, VacuumTests 26/26.

One more change follows in a separate commit: DynamicCollection.RebuildFullTextIndex still rolls back on failure without freeing the copy's pages, the same leak #175 fixes for typed collections.

[CaffeinatedCoder]

2df628c fixes the leak in the dynamic full-text rebuild on open.

• The new tree is filled in its own transaction. On failure, its pages, including those added by splits, are collected through that transaction before the rollback and then freed.
• The old index stays persisted as it was, so the next open retries.
• Dynamic CreateFullTextIndexAsync now uses the same fill-and-publish helpers.
• Each dynamic full-text index now keeps its own KeyFormat, so persisting the metadata after one rebuild can no longer mark another index that is still waiting for its rebuild as current.

The new test fails without the fix: 3000 documents, the last one unreadable, four failing opens with a stable file size, then a successful rebuild after the repair. Without the fix the file grows by about 8 MB per failing open. Full suite 2500 / 0 / 4 skipped.

[CaffeinatedCoder]

Restacked on #175's updated head (48a8145, which includes main at f8f437b), so this still has to be merged after #175. The only conflict was in CollectionSecondaryIndex.CollectAllPages, which now keeps both the R-tree branch from #205 and the full-text branch.

e67d375 uses #194: a collection with a full-text index records fulltext-index ("full-text indexes (BLite 5.3 or later)") at Open level. A version without full-text reads the index as a B-tree, and its planner would answer lookups on the property from the postings, so even read-only isn't safe. That matches the index-type rule for type 6.

• Recording: the feature is recorded whenever the index list is saved, and removed with the last full-text index. That covers typed UpdateMetadata (so CreateFullTextIndexAsync, model-declared indexes, rebuilds and drops) and dynamic PersistIndexMetadata.

• Re-recording on open: a read-write open records it again if a 5.2.x rewrite dropped the section; a read-only open writes nothing.

• Comments: the CollectionFeatures/ReadCollectionMetadata comments now say that the index-type rule only holds while the old writer keeps the index entry, and that new per-index data goes only in trailing sections. This overlaps with the same comments in fix(storage): refuse to overwrite a collection record this version cannot parse (follow-up to #194) #219 (the feat(storage): per-collection required features, refused by versions that lack them (#187) #194 follow-up). Whichever of the two lands second, I'll resolve the conflict there.

• Release note: CreateFullTextIndexAsync documents "do not open a database with full-text indexes for writing with BLite < 5.3". The same line belongs in the release notes.

• Tests:

  • typed, dynamic and model-declared create, drop and reopen;
  • re-recording after an older release dropped the section;
  • an older reader refusing even read-only, from the feature alone and from the index type alone.

  Each fails when its piece is removed; the typed read-only guard has no dedicated test.

Full-text 39/39, RequiredFeatures 11/11; full suite 2571 / 0 / 4 skipped. That clean run used xUnit.MaxParallelThreads=2, because other suites were running on the machine. Default-parallel runs only failed with lock timeouts, and those tests pass in isolation.