fix(index): split B-Tree nodes by size and find duplicates across splits (#217, #213)

[mrdevrobot]

Fixes #217. Fixes #213.

One PR for both issues, because the fixes touch the same split and descent code in BTreeIndex. The on-disk node format is unchanged.

Root causes

• B-Tree nodes split by entry count, not size: long keys overflow a node page mid-insert #217. A node split only when it reached 64 entries; nothing checked the bytes left in the page. With variable-length keys, a page filled up first and the next insert overflowed it (ArgumentOutOfRangeException). Borrow and merge on delete had no byte checks either.
• DynamicCollection non-unique index: forward scan misses duplicate keys that straddle a leaf split #213. A leaf split promotes the first key of the right half, and copies of that key can stay in the left leaf. Descent went right whenever the key equalled a separator, so forward Range, ScanRange, TryFindFirst, TryFind, Delete and the cursor's Seek all started past the left copies.
• A second bug behind DynamicCollection non-unique index: forward scan misses duplicate keys that straddle a leaf split #213, found by the randomized test. After a split, InsertIntoParent placed the promoted separator by key order. When several separators were equal, it landed after the wrong child, so the tree's order no longer matched the leaf chain. It is now placed right after the child that split.

Fix

Size-aware splits. A leaf or internal node splits when the new entry doesn't fit by count (64) or by bytes.

• The split point is computed over the existing entries plus the new one, so both halves are known to fit before anything is written. Among the points where both halves fit, it takes the best byte balance; with equal-size keys that is the old count midpoint.
• The leaf split writes the new entry straight into its half.
• An internal split sends its middle entry up to the parent, which splits recursively if needed.
• The count-only SplitNode, SplitLeafNode, SplitInternalNode and InsertIntoInternal are removed.

Key size limit. MaxKeySize is now half of a node's entry space, PageSize/2 − 36: 8156 bytes on 16 KB pages, 4060 on 8 KB. This bound guarantees that any full node can be split in two halves that each fit a page.

• The collections already reject larger keys with a BLiteValidationException naming the limit, and BTreeIndex.Insert now also throws an ArgumentException that names it.
• A tree that already holds a key above the new limit throws a clear InvalidOperationException when such a node can't be split. Per InsertAsync / InsertBulkAsync / UpsertAsync roll back the caller's explicit transaction on any exception #185/fix(collections): don't roll back a caller-owned transaction on a failed write (#185) #204, the transaction then rolls back.

Delete. Borrow and merge now check that the moved entries and the new separator fit; if they don't, the node is left underfull, which is still a valid tree. This means an internal node can end up with only its leftmost child, and a leaf can end up empty:

• merging is skipped under a parent with no separators;
• the cursor (Seek, MoveNext, MovePrev, MoveToFirst, MoveToLast) skips empty leaves, as range scans already did.

Descent. FindLeafNode and FindChildNode take a leftmost flag.

• Forward scans, TryFind/TryFindAsync and cursor Seek go left on equality and then walk the leaf chain. This costs one extra page read only when the key equals a separator or falls past a leaf's last key.
• Backward scans and inserts still go right on equality, and the unique-index check on insert is unchanged.
• Delete searches every child that can hold the key, leftmost first, and returns the leaf with the exact (key, location) and its path for rebalancing.

Page writes still go through the transaction, page reads stay WAL-aware, and a failed insert still rolls the transaction back.

Tests

BTreeSplitTests has 19 test cases:

• the B-Tree nodes split by entry count, not size: long keys overflow a node page mid-insert #217 repro on DynamicCollection (padding N = 100/400/2000) and on a typed collection (400/2000, with a reopen);
• an oversized key rejected through the collection and directly on the B-Tree;
• maximum-size keys mixed with small ones, so the split has to go by bytes;
• the DynamicCollection non-unique index: forward scan misses duplicate keys that straddle a leaf split #213 p70 repro forward and backward, a sweep over every duplicated key, and the delete of duplicates across a split, at collection and B-Tree level;
• 400 copies of one key spread over many leaves, all found and then deleted;
• a randomized test (6 seeds, 8 KB pages, keys up to near the limit, about half deleted, then more inserts) checked against a sorted in-memory multiset. It covers Range in both directions, per-key ranges, TryFind, CountRange, equality and full cursor walks, before and after a reopen. It was also run with 80 extra seeds during development.

Before the fix, 14 of the first 16 cases failed.

The two Failure_After_The_First_Page_Write... tests in ExplicitTransactionFailureTests used the #217 page overflow as their mid-write failure, which no longer happens. They now corrupt the index root page inside the caller's transaction, so the insert fails after the data page and the primary index were written, and the rollback discards the corruption.

Full BLite.Tests suite, after merging the current main into the branch: 3023 passed, 4 skipped, 0 failed. BLite.Core builds with no warnings.

Unchanged keys on update (follow-up, facc6d2)

An update whose index key does not change no longer deletes and re-inserts the entry:

• If the document stays in its slot, the entry is left alone.
• If the document moves, the entry is re-pointed with the new BTreeIndex.Relocate(key, oldLocation, newLocation), which rewrites its 6-byte location in place.

To make the first case the common one, DynamicCollection now overwrites a document that is not larger than before in its own slot (TryUpdateInPlace), as typed collections already do, and zeroes the freed tail. Typed collections re-point their secondary and primary entries the same way when a document moves.

Without this, the multi-leaf delete search that #213 needs made every update of a document with a heavily duplicated value in a DynamicCollection index read every leaf holding that value. The review measured 20,000 copies of one key deleted in random order at 5.9 s, against 0.12 s with distinct keys. Updates that grow and move a document, and deletes, still pay that search. The structural fix, unique dynamic keys with a key-format change, is tracked in #234 for v6.

Tests: UnchangedIndexKeyUpdateTests (8 tests). One checks that 5000 same-size updates of documents sharing one indexed value leave every page of that index byte-identical.

Behaviour changes

• ⚠️ Breaking: smaller key limit. MaxKeySize drops from PageSize − 64 to PageSize/2 − 36 (16 KB pages: 16320 → 8156 bytes; 8 KB pages: 8128 → 4060). That bound is what guarantees any full node can be split. Index keys and ids between the two sizes were accepted before, but broke the first time their node split; they are now rejected up front with a BLiteValidationException naming the limit. Documents already stored with such a key stay readable and deletable, and stay updatable as long as that key is unchanged: the size check is skipped for a key equal to the stored one, and its entry is kept or re-pointed. A new or changed key over the limit is rejected.
• Underfull nodes. When two nodes don't fit in one page, they are left underfull rather than redistributed by bytes, so space reuse after deleting large keys can be poor.
• Deleting a heavily duplicated key in a DynamicCollection. The delete visits every leaf holding that key until it finds the exact entry, which is linear in the number of copies. Before, it often failed silently. Typed indexes are unaffected, because their keys are unique.

🤖 Generated with Claude Code

[mrdevrobot]

Review (independent agent, two passes)

First pass on 02c2896: no correctness bug in the split, descent or rebalance logic.

• A randomized probe ran 480 times: page sizes 4/8/16 KB, four key mixes (mixed sizes, tiny vs maximum-size keys, a pool of 3 heavily duplicated keys, 40 keys), and 40 seeds each. Each run did 2500 interleaved inserts and deletes, then a rolled-back batch of inserts, then a reopen. All 480 passed.
• After each run it checked:
  • structure: every node fits its page and holds at most 64 entries; separators are sorted; every child's keys lie within its separators; all leaves are at the same depth; the leaf chain matches the in-order leaves;
  • lookups, per key against a model: Range in both directions (sync and async), CountRange, TryFind, TryFindFirst, cursor walks both ways, and Seek.
• Byte accounting is correct, including the bound that guarantees a full node can always be split at MaxKeySize. Recursive and root splits, the separator placement and the modified ExplicitTransactionFailureTests were checked too.

It raised two points, both handled in facc6d2 as decided by the maintainer:

1. Deleting copies of a heavily duplicated key in a DynamicCollection is linear per delete. Mitigated: unchanged keys are no longer deleted and re-inserted on update. The structural fix, unique dynamic keys, is DynamicCollection secondary index keys are not unique: deleting entries of a heavily duplicated value is quadratic #234 (v6).
2. The lower MaxKeySize blocked updates of documents holding an older, larger key. Fixed: unchanged keys are exempt from the size check and keep their entry. New or changed oversized keys are still rejected.

Second pass on facc6d2: nothing wrong.

• TryUpdateInPlace matches the typed in-place update. Free-space accounting stays consistent, the write goes through the transaction, CDC still emits a single Update event, and the freed tail is zeroed.
• Relocate matches the exact (key, location) entry.
• No changed key can slip past the size or unique checks.

The one missing test case it pointed out, a legacy oversized typed secondary key whose document moves, was added in 2c0851c.

Known limit, also on main: a page whose documents only shrank keeps its gaps until a delete happens on it, as with typed collections.

Found along the way, already on main, not part of this PR: a B-Tree merge frees the emptied page outside the transaction, so rolling back deletes corrupts the index. Tracked as #233 (P0).