fix(index): enforce unique secondary indexes on typed collections (#176)

[CaffeinatedCoder]

Fixes #176.

Problem

CollectionSecondaryIndex stores every B-Tree entry as [user key][document id]. Two documents with the same value therefore never produce equal keys, so BTreeIndex's unique check never fires. Typed collections accepted duplicates in indexes declared unique: true, while DynamicCollection, which stores the raw value, rejected them.

Fix

This uses the range-seek approach agreed in the issue. The composite key format stays the same.

• CollectionSecondaryIndex.EnsureUnique range-seeks [user key][min id] .. [user key][max id] on a unique B-Tree index. It throws InvalidOperationException("Duplicate key violation for unique index '<name>'") if an entry belongs to a different document. The document's own entry is skipped, so an update that keeps its value, or changes only other fields, still passes.
• CollectionIndexManager.EnsureUniqueInAll runs that check across all indexes.
• DocumentCollection calls it in InsertDataCore and UpdateAtLocationCore. Each call comes before any data page, primary-index or secondary-index write, so a rejected insert or update leaves nothing behind. Every insert, bulk insert, update and upsert goes through these two methods.
• Null keys aren't indexed, so they never conflict, which matches Insert.

Prefix handling. The seek range also contains entries of longer values that begin with the checked value. For example, "abc" lies inside the range for "ab", because the composite key has no separator. A hit counts as a duplicate only if the bytes after the user key look like an id key. They must begin with the id key's prefix byte, and for fixed-size id types (ObjectId, Guid, int, long, double, decimal) they must also have the id's exact length. That check is exact for fixed-size ids. With string ids, a value that continues with a 0x02 byte (a control character) could in theory be mistaken for a duplicate. Ruling that out would need a change to the key format, which the issue asked to keep.

Existing data. Only new writes are checked. A database that already contains duplicates in a unique index still opens and reads normally. Whether a rebuild should report existing duplicates is left for a follow-up, as agreed.

Concurrency. Within a process, the collection write lock already serializes inserts and updates. The check reads through the writing transaction, so it also sees that transaction's own uncommitted entries, including earlier documents in the same bulk insert. I haven't changed anything for multi-process access.

Tests

UniqueIndexTests adds 9 tests. The 4 rejection tests fail on main. The other 5 pass on main and guard against false positives.

• The two tests from the issue: a typed insert of a duplicate is rejected and leaves one document, and the dynamic case is rejected.
• UpdateAsync moving a document onto an existing value is rejected, and the document and its index entry are unchanged.
• A duplicate inside a bulk insert rejects the batch.
• With string ids, a duplicate is rejected, while "ab" next to "abc" is allowed.
• Not rejected: an update that keeps its value, moving a value away and reusing it, reinserting after a delete, values that share a prefix, and multiple null values.

Full suite on macOS (net10.0): 2401 passed, 4 skipped, 0 failed. BLite.Core also builds for netstandard2.1. I couldn't run BLite.NetStandard21.Tests locally because it needs a .NET 6 runtime.

Behaviour change

Inserts and updates that would duplicate a value in a unique secondary index on a typed collection now throw InvalidOperationException. Before, they succeeded silently.

Merges cleanly with #169, #170, #172, #173, #174, #175, #177 and #180.

🤖 Generated with Claude Code

[mrdevrobot]

Thanks, the fix is correct and minimal. The check runs before any write on every insert, bulk, upsert and update path. It sees the transaction's own uncommitted entries and skips the document's own entry. The id-length filter is exact for fixed-size ids, and the cost in the normal case is within noise (20k inserts: 1,045 to 1,070 ms on main, 1,050 to 1,350 ms here). 4 of the 9 new tests fail on main, full suite 2401 / 0 / 4 skipped. One change before merging.

1. A document that already shares a value can no longer be updated at all (DocumentCollection.cs:2789). UpdateAtLocationCore runs EnsureUniqueInAll(entity) on every unique index even when that index's key hasn't changed. If a database already holds duplicates, from before this fix or from CreateIndexAsync(unique: true) backfilling over them, then any update to either document throws, including a no-op UpdateAsync(e1) or a change to another field. That contradicts "only new writes are checked". Pass the old entity and check an index only when its key changed, like the short-circuit in CollectionSecondaryIndex.Update. That also saves the seek on most updates. Please add a test for it.

Worth knowing, not for this PR:

• InsertAsync, InsertBulkAsync and UpsertAsync roll back the caller's explicit transaction on any exception (~L2026, ~L2084, ~L2960); UpdateAsync only does that when it owns the transaction. A duplicate inside an explicit transaction therefore aborts the whole transaction, and the next CommitAsync fails with "state Aborted". Please open an issue.
• Two concurrent explicit transactions inserting the same value both commit, and one insert is lost. The same happens without any index, so it's an isolation problem, not a uniqueness one.
• The seek range [key][]..[key][FF×20] also walks every longer value starting with the key. With 11,110 values starting with "1", inserting "1" took about 20 ms. Starting the range at [userKey][id prefix byte] would skip them.
• Typed collections allow any number of nulls; DynamicCollection indexes an explicit BSON null and rejects a second one. Worth a sentence in the docs.

Nits: cache whether the collection has any unique B-Tree index so EnsureUniqueInAll can skip the loop; the ownIdKey.Data.Length > 0 guard is dead code; the XML doc says "before writing anything", but on update it runs after reading the old document.

[CaffeinatedCoder]

Thanks. Addressed in e369cb1.

1. Updates of documents that already share a value. EnsureUniqueInAll now takes the stored document. On update, an index is checked only when its key changed: Equals(old, new) as in CollectionSecondaryIndex.Update, or byte-equal encoded keys (which also covers byte[] keys). New test Typed_Unique_Index_Allows_Updates_Of_Documents_That_Already_Share_A_Value:

• inserts two duplicates, then CreateIndexAsync(unique: true) backfills over them;
• a no-op UpdateAsync, an UpsertAsync and an update of another field all succeed;
• moving the key onto a third document's value still throws and leaves the document unchanged.

It fails when null is passed instead of the old entity.

Seek range. Done in the same commit. Every id key starts with its prefix byte (0x02, or 0x03 for decimal), so the scan now runs from [userKey][p] to [userKey][p+1]. Most of the gain comes from the upper bound, since "10", "11", … sort after [02 31 02 …]. With 11,110 values starting with "1", inserting "1" went from 2–80 ms to about 0.1 ms here. Typed_Unique_Index_Separates_Values_Followed_By_Low_Control_Characters covers values followed by 0x01, 0x02 and 0x03 at the new bounds. It only guards correctness; the change itself only affects speed.

There is one remaining false positive, and the old range had it too. With string ids, an existing value equal to the checked value followed by U+0002 is reported as a duplicate, because "ab\u0002" + id "1" and "ab" + id "\u00021" encode to the same composite bytes. Fixing that needs a key-format change, so I left it.

Nulls. I added a sentence under "Index Management" in the README, and Dynamic_Unique_Index_Rejects_A_Second_Explicit_Null pins down the DynamicCollection behaviour.

Nits.

• CollectionIndexManager caches whether any unique B-Tree index exists (refreshed on create, drop, recreate and dispose), and EnsureUniqueInAll returns early when there is none.
• The dead ownIdKey.Data.Length > 0 guard is removed.
• The XML docs now say the check runs before any page is written; on update, that is after the old document has been read.

Transaction abort: filed as #185. I left the concurrent-transactions lost insert out; as you say, it's isolation rather than uniqueness.

Full suite: 2404 passed, 0 failed, 4 skipped.

[mrdevrobot]

Verified e369cb1. With a unique index created over existing duplicates, a no-op update, an update of another field, and updates of the other duplicate all succeed; changing the key onto a taken value throws, onto a fresh value succeeds; the same holds in an explicit transaction and for upsert. int, decimal (1.5m vs 1.500m unchanged, 3.00m conflicts with 3m), byte[] and compound keys behave correctly, and string ids starting with 0x01–0x03 give no false negatives. The new range also fixes the prefix cost: inserting "1" next to 11,110 values starting with "1" went from 8–18 ms to about 0.1 ms. Full suite on main with #169/#170 merged: 2422 / 0 / 4 skipped.

The string-id false positive you disclosed is a bit broader than "ab\u0002": any longer value whose next byte is 0x02 blocks the shorter one, e.g. an existing "ab\u0002xyz" blocks inserting "ab". It needs a key-format change; fine to track separately.

Approving and merging.