feat(storage): per-collection required features, refused by versions that lack them (#187)

[CaffeinatedCoder]

Closes #187.

This adds the per-collection "required features" value discussed in #177. A collection records the features it depends on that older versions cannot maintain. A version that finds a feature it does not know refuses the collection instead of silently damaging it. The page format version stays unchanged, as agreed, so older versions still open the database. The PR defines no feature yet; full-text indexes (#177) will be the first. It is meant to be merged before any release that ships #177, so that every release from then on can refuse what it does not know.

Format

The record gets a new section after the index key formats, the last section today:

int32  count
count × { string id; byte requirement; string description }

• id is a stable identifier that versions implementing the feature recognise, e.g. fulltext-index.
• description is stored with the feature, e.g. full-text indexes (BLite 5.3 or later), so that a version that has never heard of the feature can still say what is missing and which version to use.
• Compatibility in both directions:
  • Records without the section load with no features.
  • Releases that stop reading at the key formats load the record unchanged.
  • New sections go after this one.

Requirement levels

requirement	Meaning	Version without the feature
Open (1)	A version without the feature would read the collection wrongly, e.g. serve queries from an index it misreads (the #177 planner issue).	Refuses to open the collection.
Write (2)	Reads are correct; writes would leave something out of date.	Refuses unless the database is opened read-only (PageFileConfig.Access = Read), then opens it for reading.
anything else	Reserved.	Treated as Open.

I added both levels now rather than starting with one. A second level added later would be ignored by every version released in between: those versions would write to a collection that only needs to be protected from writers. So the distinction has to exist in the first version that checks.

Where it is enforced

• Opening a collection. Both the typed path (CollectionIndexManager, so before DocumentCollection writes anything) and the dynamic path (DynamicCollection) throw NotSupportedException. The message names the collection, this version and every missing feature:

  Collection 'articles' requires features that BLite 5.2.3 does not support: full-text indexes (BLite 5.3 or later). Use a BLite version that supports these features.

  A DocumentDbContext that fails this way disposes its storage, as with any other open failure.

• Writing metadata outside an open collection. SaveCollectionMetadata and DeleteCollectionMetadata check the record that is already stored, not the object passed in, and refuse to overwrite or drop it. FreeCollectionPages refuses too, because it can't find the pages of an index type it doesn't know. Nothing is written before these checks.

• DocumentDbContext orphan cleanup. An unmapped collection that has unknown features is no longer dropped at open. It belongs to the newer version that created it, and dropping it would also leak the pages this version can't find.

• BLiteEngine:

  • ListCollections lists such a collection without opening it. Before, one such collection would have made the whole listing throw.
  • VacuumAsync() over all collections refuses before vacuuming anything, since it could not erase that collection's deleted documents.
  • VacuumAsync with that CollectionName throws instead of returning silently.
  • TruncateCollectionAsync, migration (BLiteMigration) and GDPR subject export throw when they open it, rather than skipping a collection they can't read.

• Index types. An index whose type is above the highest IndexType this version defines counts as an unknown Open feature (type 0 stays accepted). This keeps a new index type protected even after a release without this check has rewritten the record and dropped the section. When feat(index): full-text search with BM25 ranking (#70) #177 adds FullText = 6, the limit moves with the enum.

Limitations

• Releases before this PR can't refuse. They also drop the section when they rewrite a record. I checked with BLite.Core 5.2.3 from NuGet (details below). The index-type rule covers index features in that case. The "do not open with BLite < 5.3 for writing" note from feat(index): full-text search with BM25 ranking (#70) #177 stays.
• Multi-process: a process that already has the collection open doesn't notice a feature that another process adds afterwards. This is the same class of problem as Multi-process mode: two processes opening a new model both backfill the same index #192.
• Database-wide formats (page format, key dictionary, WAL) are not covered. This is per collection, as discussed.

Plan for #177

Once this is merged, #177 would:

1. add fulltext-index to the known features;
2. record new RequiredFeature("fulltext-index", Open, "full-text indexes (BLite 5.3 or later)") whenever a collection has a full-text index, and remove it when the last one is dropped (typed UpdateMetadata, dynamic PersistIndexMetadata);
3. add tests for setting and clearing it.

The two cases you listed in #177 (an older DynamicCollection that drops the type-6 index from the metadata, and an older typed version that leaves stale postings on delete) are then refused by any release that has this PR. The branches merge without conflicts. On the merge, #177's full-text tests and these pass, and the index-type rule accepts type 6 as known.

Tests

RequiredFeaturesTests, 11 tests. A newer version is simulated by adding the feature to the stored metadata.

• Persistence: the round trip keeps the id, the requirement (including an unknown value) and the description. A record cut back to the pre-section layout loads with no features and keeps every other field. The collection stays usable, and the next save writes an empty section.
• Typed: the context refuses with a message naming the collection and the feature, and the files are released. An unmapped collection with an unknown feature is left alone, while a plain orphan is still dropped.
• Engine:
  • opening, TruncateCollectionAsync and named vacuum throw;
  • ListCollections lists it;
  • vacuum over all collections refuses;
  • other collections stay readable and writable;
  • the record and pages are unchanged afterwards.
• Export and migration: a GDPR subject export over all collections and BLiteMigration.ToMultiFileAsync throw instead of skipping the collection, and the source database is left unchanged.
• Storage: SaveCollectionMetadata, FreeCollectionPages and DeleteCollectionMetadata throw and leave the record unchanged.
• Levels:
  • a Write feature is refused read-write, with the message pointing to read-only, but opens and reads read-only;
  • an unknown level is refused even read-only and is named by its id when there is no description;
  • a message lists every unsupported feature.
• Index types: an index of type 200 without any recorded feature is refused for typed and dynamic opens and is still listed.

Each check was disabled in turn: the open checks, the listing, orphan cleanup, the save/delete/free guards, named vacuum, the read-only flag, the unknown-level handling and the index-type rule. Each time, the matching tests fail.

Full suite on macOS: 2403 passed, 0 failed, 4 skipped. BLite.NetStandard21.Tests: 20/20. BLite.Core builds for net10.0 and netstandard2.1 with no warnings.

Checked against BLite.Core 5.2.3 (NuGet)

I used two scratch console apps, not part of the PR, one on this branch and one on the published 5.2.3 package:

Scenario	Result
This branch writes a collection and records a feature; 5.2.3 opens it, lists, reads, inserts and creates an index	Works. 5.2.3 ignores the section. Its metadata rewrite drops the section (the known limitation).
5.2.3 creates a database; this branch opens, reads and inserts; 5.2.3 opens it again	Works both ways. 5.2.3 reads the record with the (empty) section.
This branch records an unknown feature, then opens the collection	Refused with the message above.

Cost

The only per-operation cost is in SaveCollectionMetadata, which now parses the one stored record it has already located. 20,000 GetNextSequence calls (each a metadata read and save) took 7.38 s on this branch and 7.21 s on main, and later rounds were within noise. Both grow with every round because each save adds a slot to the metadata chain; that is independent of this PR.

Release notes (suggestion)

Collections can now record features they depend on. From this release on, a BLite version that does not know such a feature refuses to open or modify the collection, with an error naming the feature and the version that supports it, instead of damaging it. Releases before this one cannot refuse.

🤖 Generated with Claude Code

[mrdevrobot]

Thanks, this is the right design. I checked that 5.2.x reads the collection record with a stream bounded by the slot length and stops after the key formats, so the trailing section is ignored; and I reproduced the compatibility table against NuGet 5.2.3. Refusal is per collection, before anything is written, and read-only opens accept Write-level features. With each check disabled in turn, 8, 1 and 1 of the 11 tests fail. Merged with main: BLite.Core builds with no warnings on both targets, full suite 2468 / 0 / 4 skipped, NetStandard21 20/20.

Approving. Three things to handle, none blocking:

1. Conflict with #196. Both rewrite SaveCollectionMetadata. Git places this PR's guard after #196's new in-place write, where it no longer covers that path; with the guard there, Storage_RefusesToRewriteDeleteOrFree_ACollectionWithAnUnknownFeature fails. Whichever lands second should put the check right after FindMetadataRecord, before the in-place step, e.g. if (foundPage >= 0 && TryReadRecord(chain[foundPage].Buffer, ReadSlot(...)) is {} stored) EnsureCanWrite(stored);. Resolved that way, main + #194 + #196 passes the full suite (2474 / 0 / 4 skipped). I'd merge #196 first: its up-front size check also fixes the pre-existing case where an oversized record fails after the old one was marked deleted.
2. The description overstates the index-type rule. 5.2.3's DynamicCollection.PersistIndexMetadata clears Indexes and rebuilds the list from the indexes it loaded, so a dynamic CreateIndexAsync or DropIndex in 5.2.3 drops both the feature section and the type-6 entry. I reproduced it: after a 5.2.3 dynamic CreateIndexAsync, features = 0 and this branch opens the collection without refusing. The rule only helps when the old writer keeps the index entry. Please reword the description and the CollectionFeatures remarks, and keep the "don't open with BLite < 5.3 for writing" release note from #177.
3. Future index types. If a later index type adds per-index bytes inline in the index list (as Vector does with dims/metric), this version misreads the record, GetCollectionMetadata returns null, and the dynamic open creates a fresh record, so the refusal never runs. Please add the rule "new per-index data goes only in trailing sections" to the ReadCollectionMetadata and CollectionFeatures comments.

Nits: ListCollections now returns names that GetOrCreateCollection throws on, worth a release-note line; TryReadRecord swallows every exception, so an unparsable record skips the guard (consider refusing); check that the release pipeline stamps the assembly version used in the message (5.3.0, not 1.0.0).

[mrdevrobot]

#196 is now merged (4250e29), so this branch conflicts in StorageEngine.Collections.cs as expected. Please rebase onto main and resolve it as described in my review: run the feature check right after FindMetadataRecord, before #196's in-place write, so it also covers that path. Storage_RefusesToRewriteDeleteOrFree_ACollectionWithAnUnknownFeature fails if the guard ends up after the in-place step. I'll merge once CI is green on the rebased branch.

[mrdevrobot]

I rebased this onto main (46918ee) since #196 is in. The only conflict was SaveCollectionMetadata in StorageEngine.Collections.cs. The feature check now runs right after FindMetadataRecord, before #196's in-place write, so it covers every write path. #196's save, compaction and TryReadRecordName are kept as on main. TryReadRecord now takes a SlotEntry. With the check disabled, Storage_RefusesToRewriteDeleteOrFree_ACollectionWithAnUnknownFeature fails; with it, the 11 RequiredFeaturesTests pass. BLite.Core builds with no warnings and the full suite gives 2480 / 0 / 4 skipped. Your authorship is kept on the commit. If you pull this branch locally, reset to the new head first.

[CaffeinatedCoder]

The review follow-ups are in #219. A stored record that doesn't parse is now refused instead of overwritten. The CollectionFeatures remarks now describe the index-type rule accurately (5.2.3's dynamic PersistIndexMetadata drops the entry) and add the trailing-sections-only format rule. The release-note text is in that PR's description. The guard placement you used when merging with #196 is unchanged; EnsureCanReplaceRecord sits in the same spot.