Skip to content

fix(storage): refuse to overwrite a collection record this version cannot parse (follow-up to #194) - #219

Merged
mrdevrobot merged 1 commit into
EntglDb:mainfrom
CaffeinatedCoder:fix/194-review-followups
Oct 15, 2026
Merged

mrdevrobot merged 1 commit into
EntglDb:mainfrom
CaffeinatedCoder:fix/194-review-followups

Conversation

@CaffeinatedCoder

Copy link
Copy Markdown
Collaborator

Follow-up to the #194 review (#187).

Changes

  • Records that don't parse are refused. SaveCollectionMetadata and DeleteCollectionMetadata now throw NotSupportedException when the collection's stored record cannot be parsed. The message names the collection and this version, and the record is left unchanged. Before, TryReadRecord swallowed every exception, so the feature check was skipped and the record was overwritten. Such a record also makes GetCollectionMetadata return null, so a dynamic open saved a fresh, empty record over it.
    • EnsureCanReplaceRecord replaces TryReadRecord in the same place: right after FindMetadataRecord, before the in-place step.
    • The delete guard now finds the record with LoadMetadataChain/FindMetadataRecord instead of GetCollectionMetadata, which skips records it cannot parse.
  • Index-type rule, documented accurately (CollectionFeatures remarks). The rule only protects a new index type if the older release that rewrote the record kept the index entry. Typed collections and saves that start from the stored record keep it. 5.2.3's DynamicCollection.PersistIndexMetadata doesn't: it rebuilds the index list from the indexes it loaded, on every dynamic index create/drop and on a vacuum with RebuildIndexes, and so drops the entry together with the feature section.
  • Format rule (CollectionFeatures and ReadCollectionMetadata): new per-index data goes only in trailing sections, never inline in the index list. An older version reads the list by position. Unexpected bytes there make the record fail to parse; GetCollectionMetadata then returns null, a dynamic open saves a fresh record, and the feature refusal never runs.
  • Version in the refusal message: checked. versionize bumps <Version>, and nuget-publish.yml passes -p:Version=<tag>. A build with -p:Version=5.3.0-beta.1 has assembly version 5.3.0.0, so the message says "BLite 5.3.0".

Tests

RecordThisVersionCannotParse_IsNeitherOverwrittenNorDropped: a record is cut right after its name.

  • SaveCollectionMetadata and DeleteCollectionMetadata throw.
  • GetOrCreateCollection throws.
  • Another collection stays readable and writable.
  • The stored bytes are unchanged.

The test fails with main's guard, and also with only the Delete guard reverted. RequiredFeaturesTests + MetadataChainTests: 18/18. BLite.Core builds for net10.0 and netstandard2.1 with no warnings. Full suite: 2519 passed; the 5 failures were lock timeouts under heavy parallel load, and each passed when rerun on its own. BLite.NetStandard21.Tests couldn't run locally (it needs the net6.0 runtime).

Release notes (suggestion)

Collections can now record features they depend on. From this release on, a BLite version that does not know such a feature refuses to open or modify the collection, with an error naming the feature and the version that supports it, instead of damaging it.

Do not open a database with BLite < 5.3 for writing once it has been used by 5.3 or later. Older releases cannot refuse, and they drop the recorded features when they rewrite a collection's metadata. In those releases, a dynamic index create or drop, or a vacuum that rebuilds indexes, also drops index entries of types they don't know.

BLiteEngine.ListCollections now also lists collections this version cannot open; GetOrCreateCollection throws NotSupportedException for them.

A collection whose stored metadata this version cannot parse is no longer overwritten or dropped; opening or modifying it throws NotSupportedException.

🤖 Generated with Claude Code

…nnot parse (EntglDb#187)

Follow-up to the EntglDb#194 review:

- SaveCollectionMetadata and DeleteCollectionMetadata now refuse a stored
  record that does not parse, with a NotSupportedException naming the
  collection, instead of skipping the feature check and overwriting it.
  An unparsable record made GetCollectionMetadata return null, so a
  dynamic open would save a fresh, empty record over it.
  EnsureCanReplaceRecord replaces TryReadRecord; the Save guard stays right
  after FindMetadataRecord, before the in-place step. The Delete guard
  uses LoadMetadataChain/FindMetadataRecord instead of
  GetCollectionMetadata, which skips unparsable records.
- CollectionFeatures remarks: the index-type rule protects a new index type
  only while the older release keeps the index entry; 5.2.3's dynamic
  PersistIndexMetadata (index create/drop, vacuum with RebuildIndexes)
  drops the entry together with the section.
- CollectionFeatures and ReadCollectionMetadata: new per-index data goes
  only in trailing sections, never inline in the index list, and why.
- Test: a record cut after its name is neither overwritten nor dropped,
  the dynamic open refuses, and other collections stay usable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@mrdevrobot mrdevrobot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. Refusing to overwrite or drop a record this version can't parse closes the gap where TryReadRecord swallowed the error and a dynamic open saved an empty record over it. The format rule in CollectionFeatures and ReadCollectionMetadata is a good thing to have written down. Note for the release notes: a collection whose metadata record is damaged can now no longer be opened or dropped either; the message says so, which is fine. Verified together with #218 and #215 on top of main: full suite green.

@mrdevrobot
mrdevrobot merged commit d7f0dd7 into EntglDb:main Oct 15, 2026
2 checks passed
mrdevrobot added a commit to CaffeinatedCoder/BLite that referenced this pull request Oct 15, 2026
Conflict with EntglDb#219 in DeleteCollectionMetadata resolved by keeping
main's EnsureCanReplaceStoredRecord, which reads the chain directly and
so does not re-enter _metadataLock.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mrdevrobot added a commit to CaffeinatedCoder/BLite that referenced this pull request Oct 15, 2026
EntglDb#175 and EntglDb#219 landed as squashes. Conflicts resolved:
- DocumentCollection, CollectionIndexManager: this branch's side
  (TargetKeyFormat, ResolveIndexName by index type, ApplyIndexBuilder ct).
- CollectionFeatures remarks: main's wording from EntglDb#219, which covers the
  same index-type and format rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mrdevrobot added a commit to CaffeinatedCoder/BLite that referenced this pull request Oct 15, 2026
…remarks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mrdevrobot added a commit that referenced this pull request Oct 15, 2026
* fix(index): fill model-declared indexes from existing documents (#171)

An index declared in OnModelCreating was created empty when the collection
already held documents, so queries routed to it missed every older
document. This hit the usual way of adding an index: a new app version that
adds HasIndex, HasVectorIndex or HasSpatialIndex to an existing collection.

ApplyIndexBuilder now builds a missing index unregistered, fills it from the
primary index, and only then registers and persists it. A crash before that
leaves no empty index in the metadata, and the next open builds it again.
If a document cannot be loaded or indexed, the open fails with an
InvalidOperationException naming the index, and nothing is persisted.
Existing indexes are still a cheap name lookup on every open.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(index): address review of model-declared index backfill (#171)

- The error when an existing document cannot be indexed now says that
  nothing was saved and that every open fails the same way until the
  documents can be indexed or the index is removed from OnModelCreating,
  and includes the underlying message.
- Removes CollectionIndexManager.EnsureIndexUntyped and the public
  CreateVectorIndexUntyped / CreateSpatialIndexUntyped, which BuildDefinition
  made dead code.
- The CreateUnregistered doc comment notes that a spatial index cannot
  release its pages yet.

Opening a context whose model declares a new index over documents it
cannot index now throws; before, the index was created empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(index): address follow-ups on model-declared index backfill (#171)

- Restore the public CreateVectorIndexUntyped and CreateSpatialIndexUntyped
  on CollectionIndexManager, unchanged but marked [Obsolete]; the internal
  EnsureIndexUntyped stays removed.
- The backfill failure message ends the inner message with exactly one
  full stop before "Nothing was saved".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(index): free a partially built model index and roll back a failed registration (#171)

Address review of #175 on top of #170:

- ApplyIndexBuilder fills the new index through the same helper as the
  vacuum rebuild (FillUnregisteredIndexAsync): its own transaction, pages
  collected through it before the rollback, then freed. Before, a fill
  that failed after splits freed only the root, so every failing open
  leaked the rest of the partial tree and grew the file.
- PublishFilledIndex hands a filled index to Register or ReplaceIndex and
  frees it if it did not get registered; used by both callers.
- CollectionIndexManager: Register and ReplaceIndex share PublishLocked,
  which restores the previous registration (or none) when the metadata
  cannot be saved, and refreshes the unique-index flag. Register skipped
  that refresh, so a unique index added by the model was not enforced
  until the next open.
- CreateUnregisteredCopy is gone; callers use CreateUnregistered with the
  existing definition, which now checks for disposal.

Tests: a fill that fails on the last of 3000 documents does not grow the
file over repeated opens; a failed metadata save leaves the index
unregistered and frees its pages; a unique model index is enforced in the
session that adds it. All three fail without these changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(index): migrate legacy decimal indexes into a copy and free it on failure (#171)

MigrateLegacyDecimalIndexes replaced the index with an empty tree
(RecreateIndex) and then filled it in place. A rebuild that failed after
splits rolled back and left those pages linked from nowhere, and the next
open freed only the root of the emptied tree, so every failing open
leaked the partial tree and grew the file. The legacy tree was also gone
after the first failure.

The migration now uses the same helpers as model indexes and the vacuum
rebuild: it fills an unregistered copy (FillUnregisteredIndexAsync) set
to the current key format and swaps it in with ReplaceIndex through
PublishFilledIndex, so the new format and root are one metadata save.
On failure the copy is freed and the old index stays registered in its
old format with its data, and the next open retries.

RecreateIndex has no callers left and is removed.

Test: a legacy-format decimal index over 3000 documents whose last one
cannot be read; four failing opens do not grow the file, the stored
index keeps its root and legacy format, and the migration succeeds once
the document is repaired. Fails without this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(index): full-text search with BM25 ranking (#70)

Adds a full-text index type for string properties, stored as a companion
inverted index in one B-Tree per index:

- postings [0x01][term][0x00][location][tf][doc length] -> location, so
  BM25 scores come from the postings alone and a delete recomputes the
  exact keys from the old text;
- one statistics entry [0x02][documents][terms], kept in the same
  transaction as the postings.

Text is analyzed with TextNormalizer (accents removed, lowercased,
punctuation as separators) and split into terms of at most 32 UTF-8 bytes.

API:
- HasFullTextIndex(x => x.Text) and CreateFullTextIndexAsync(x => x.Text)
  (default name idx_fts_{property}), filled from existing documents before
  the index is registered;
- LINQ x.Text.FullTextMatch("terms"): documents containing every term,
  served from the index and evaluated with the same rules in memory;
- FullTextSearchAsync(index, query, k, matchAllTerms) returning the top k
  documents with their BM25 score;
- DynamicCollection.CreateFullTextIndexAsync / FullTextSearchAsync.

Equality, range and IN lookups no longer pick a vector, spatial or
full-text index just because it covers the property.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fts): address review of full-text search (#70)

- Tag postings 0x10 and statistics 0x20, outside the range of the generic
  [0x00..0x03] keys an older BLite writes when it reads the tree as a plain
  B-Tree; validate the statistics key length.
- Record FullTextIndex.FormatVersion as the index KeyFormat and rebuild the
  index on open (typed and dynamic) when it differs or the tree holds keys
  the index did not write, since deletes rely on reproducing the keys.
- Re-check every hit against the document text, for LINQ FullTextMatch and
  FullTextSearchAsync (typed and dynamic), so a stale posting never
  surfaces the document that now holds its location.
- Search with matchAllTerms intersects rarest-first before scoring, and
  falls back to statistics estimated from the postings when the entry is
  missing (e.g. lost to a concurrent transaction's update).
- CreateFullTextIndexAsync throws when the name belongs to a non-full-text
  index, passes its token to the rebuild, and is declared on
  IDocumentCollection; DynamicCollection.FullTextSearchAsync takes a token.
- Document the concurrent-transaction statistics caveat, batching inserts
  and the CJK / supplementary-plane limitation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(fts): batching advice and name-collision exception on CreateFullTextIndexAsync (#70)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* test(fts): a failed rebuild triggered by foreign keys is retried on the next open (#70)

A full-text index whose format is current but whose tree holds keys it did
not write (e.g. from an older BLite reading it as a plain B-Tree) is
rebuilt on open. MigrateLegacyDecimalIndexes (on #175's base) rebuilds it
into an unregistered copy and swaps that in only after it committed, so a
rebuild that fails leaves the old index registered as it was: still
holding the foreign keys, still needing the migration.

- MigrateLegacyDecimalIndexes doc comment covers full-text indexes.
- Test: corrupt a document so the rebuild on open fails; over three
  failing opens the stored root and format are unchanged, the foreign key
  is still there and the file does not grow; after repairing the document
  the next open swaps in the rebuilt index and a further open keeps it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fts): return k re-checked hits and each location once (#70)

- FullTextSearchAsync (typed and dynamic) keeps taking candidates past
  the first k, in widening windows, until k hits survive the re-check
  against the document; the first window costs one Search as before.
- Postings yields one posting per location, so a stale posting sharing
  its term and location with the current document's no longer makes
  Match return the location twice or Search score it twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(fts): rarest-first saves decoding and allocation, not I/O (#70)

CountRange walks every leaf entry of each query term, so looking up the
few candidates in a common term's postings avoids decoding them, not
reading their pages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(fts): a stale posting at a document's location can be the one scored (#70)

- Postings documents that the first posting of a location in key order
  (lowest frequency, then shortest length) is returned, which can be a
  stale one, so its frequency and length can be used for BM25 scoring,
  and CountRange counts both towards the document frequency. Matches
  and once-per-location results are unaffected.
- The test for a stale posting at a document's location notes that it
  checks membership and order, not scores.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* fix(fts): free a failed dynamic full-text rebuild and keep the old index (#70)

DynamicCollection rebuilt a stale full-text index on open into a fresh
tree and, if a document could not be read or indexed, rolled back and
threw without freeing that tree. Pages added by splits are allocated
outside the transaction, so every failing open leaked the partial tree
and grew the file. CreateFullTextIndexAsync leaked the same way.

- FillUnregisteredFullTextIndexAsync fills the new index in its own
  transaction; on failure it collects the index's pages through the
  transaction before the rollback and frees them. Used by the rebuild on
  open and by CreateFullTextIndexAsync.
- PublishFullTextIndex registers the filled index and persists the
  metadata; if that fails it restores the previous registration and
  frees the new index.
- Each full-text index keeps its own KeyFormat and PersistIndexMetadata
  writes it, so persisting after one index's rebuild no longer marks
  another index that still awaits its rebuild as current (e.g. a crash
  between two rebuilds).

Test: a dynamic full-text index over 3000 documents with a foreign key
and an unreadable last document; four failing opens do not grow the
file and keep the old index (same root, still searchable, still flagged
for rebuild), and the rebuild succeeds once the document is repaired.
Fails without this change.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* feat(fts): record full-text indexes as a required feature of the collection (#70)

Uses the per-collection required features from #194, so BLite versions
that have that check but not full-text indexes refuse the collection
instead of misreading it.

- CollectionFeatures.FullTextIndexes: id "fulltext-index", description
  "full-text indexes (BLite 5.3 or later)", requirement Open. A version
  without full-text indexes reads the index as a B-Tree: its planner
  answers equality, range and IN lookups on the property from the
  postings, so even read-only results are wrong. It is a known feature
  in this version.
- SyncIndexFeatures records the feature while the collection has a
  full-text index and removes it with the last one, leaving every other
  feature alone. Called whenever the index list is written: typed
  UpdateMetadata (CreateFullTextIndexAsync, model-declared indexes via
  ApplyIndexBuilder, rebuilds, DropIndex) and dynamic
  PersistIndexMetadata.
- A read-write open records the feature again if a release that
  predates it rewrote the record and dropped the section (5.2.x does);
  a read-only open leaves the file alone.
- CollectionFeatures and ReadCollectionMetadata document that the
  index-type rule only helps while an old writer keeps the index entry
  (a 5.2.x dynamic collection drops both), and that new per-index data
  goes only in trailing sections.
- CreateFullTextIndexAsync (typed and dynamic) notes: do not open a
  database with full-text indexes for writing with BLite < 5.3.

Tests: the feature is recorded for model-declared, typed and dynamic
indexes, survives reopen, stays while another full-text index is left
and is cleared with the last one; it is recorded again on a read-write
open after being dropped, not on a read-only one; a version without
full-text indexes (known features and index types passed in) refuses to
open the collection even read-only, from the feature alone and from the
index type alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(storage): drop the format-rule note duplicated by #219's remarks

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: MrDevRobot <mrdevrobot@entgldb.com>

This branch was successfully deployed

1 active deployment
Production — 46c0cf61 Deployed Oct 13, 2026 by CaffeinatedCoder via build_and_publish #688
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants