Repository navigation
fix(storage): refuse to overwrite a collection record this version cannot parse (follow-up to #194) - #219
Merged
mrdevrobot merged 1 commit intoOct 15, 2026
Conversation
…nnot parse (EntglDb#187) Follow-up to the EntglDb#194 review: - SaveCollectionMetadata and DeleteCollectionMetadata now refuse a stored record that does not parse, with a NotSupportedException naming the collection, instead of skipping the feature check and overwriting it. An unparsable record made GetCollectionMetadata return null, so a dynamic open would save a fresh, empty record over it. EnsureCanReplaceRecord replaces TryReadRecord; the Save guard stays right after FindMetadataRecord, before the in-place step. The Delete guard uses LoadMetadataChain/FindMetadataRecord instead of GetCollectionMetadata, which skips unparsable records. - CollectionFeatures remarks: the index-type rule protects a new index type only while the older release keeps the index entry; 5.2.3's dynamic PersistIndexMetadata (index create/drop, vacuum with RebuildIndexes) drops the entry together with the section. - CollectionFeatures and ReadCollectionMetadata: new per-index data goes only in trailing sections, never inline in the index list, and why. - Test: a record cut after its name is neither overwritten nor dropped, the dynamic open refuses, and other collections stay usable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mrdevrobot
approved these changes
Oct 15, 2026
mrdevrobot
left a comment
Contributor
There was a problem hiding this comment.
Thanks. Refusing to overwrite or drop a record this version can't parse closes the gap where TryReadRecord swallowed the error and a dynamic open saved an empty record over it. The format rule in CollectionFeatures and ReadCollectionMetadata is a good thing to have written down. Note for the release notes: a collection whose metadata record is damaged can now no longer be opened or dropped either; the message says so, which is fine. Verified together with #218 and #215 on top of main: full suite green.
mrdevrobot
added a commit
to CaffeinatedCoder/BLite
that referenced
this pull request
Oct 15, 2026
Conflict with EntglDb#219 in DeleteCollectionMetadata resolved by keeping main's EnsureCanReplaceStoredRecord, which reads the chain directly and so does not re-enter _metadataLock. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mrdevrobot
added a commit
to CaffeinatedCoder/BLite
that referenced
this pull request
Oct 15, 2026
EntglDb#175 and EntglDb#219 landed as squashes. Conflicts resolved: - DocumentCollection, CollectionIndexManager: this branch's side (TargetKeyFormat, ResolveIndexName by index type, ApplyIndexBuilder ct). - CollectionFeatures remarks: main's wording from EntglDb#219, which covers the same index-type and format rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mrdevrobot
added a commit
to CaffeinatedCoder/BLite
that referenced
this pull request
Oct 15, 2026
…remarks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
mrdevrobot
added a commit
that referenced
this pull request
Oct 15, 2026
* fix(index): fill model-declared indexes from existing documents (#171) An index declared in OnModelCreating was created empty when the collection already held documents, so queries routed to it missed every older document. This hit the usual way of adding an index: a new app version that adds HasIndex, HasVectorIndex or HasSpatialIndex to an existing collection. ApplyIndexBuilder now builds a missing index unregistered, fills it from the primary index, and only then registers and persists it. A crash before that leaves no empty index in the metadata, and the next open builds it again. If a document cannot be loaded or indexed, the open fails with an InvalidOperationException naming the index, and nothing is persisted. Existing indexes are still a cheap name lookup on every open. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(index): address review of model-declared index backfill (#171) - The error when an existing document cannot be indexed now says that nothing was saved and that every open fails the same way until the documents can be indexed or the index is removed from OnModelCreating, and includes the underlying message. - Removes CollectionIndexManager.EnsureIndexUntyped and the public CreateVectorIndexUntyped / CreateSpatialIndexUntyped, which BuildDefinition made dead code. - The CreateUnregistered doc comment notes that a spatial index cannot release its pages yet. Opening a context whose model declares a new index over documents it cannot index now throws; before, the index was created empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(index): address follow-ups on model-declared index backfill (#171) - Restore the public CreateVectorIndexUntyped and CreateSpatialIndexUntyped on CollectionIndexManager, unchanged but marked [Obsolete]; the internal EnsureIndexUntyped stays removed. - The backfill failure message ends the inner message with exactly one full stop before "Nothing was saved". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(index): free a partially built model index and roll back a failed registration (#171) Address review of #175 on top of #170: - ApplyIndexBuilder fills the new index through the same helper as the vacuum rebuild (FillUnregisteredIndexAsync): its own transaction, pages collected through it before the rollback, then freed. Before, a fill that failed after splits freed only the root, so every failing open leaked the rest of the partial tree and grew the file. - PublishFilledIndex hands a filled index to Register or ReplaceIndex and frees it if it did not get registered; used by both callers. - CollectionIndexManager: Register and ReplaceIndex share PublishLocked, which restores the previous registration (or none) when the metadata cannot be saved, and refreshes the unique-index flag. Register skipped that refresh, so a unique index added by the model was not enforced until the next open. - CreateUnregisteredCopy is gone; callers use CreateUnregistered with the existing definition, which now checks for disposal. Tests: a fill that fails on the last of 3000 documents does not grow the file over repeated opens; a failed metadata save leaves the index unregistered and frees its pages; a unique model index is enforced in the session that adds it. All three fail without these changes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(index): migrate legacy decimal indexes into a copy and free it on failure (#171) MigrateLegacyDecimalIndexes replaced the index with an empty tree (RecreateIndex) and then filled it in place. A rebuild that failed after splits rolled back and left those pages linked from nowhere, and the next open freed only the root of the emptied tree, so every failing open leaked the partial tree and grew the file. The legacy tree was also gone after the first failure. The migration now uses the same helpers as model indexes and the vacuum rebuild: it fills an unregistered copy (FillUnregisteredIndexAsync) set to the current key format and swaps it in with ReplaceIndex through PublishFilledIndex, so the new format and root are one metadata save. On failure the copy is freed and the old index stays registered in its old format with its data, and the next open retries. RecreateIndex has no callers left and is removed. Test: a legacy-format decimal index over 3000 documents whose last one cannot be read; four failing opens do not grow the file, the stored index keeps its root and legacy format, and the migration succeeds once the document is repaired. Fails without this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(index): full-text search with BM25 ranking (#70) Adds a full-text index type for string properties, stored as a companion inverted index in one B-Tree per index: - postings [0x01][term][0x00][location][tf][doc length] -> location, so BM25 scores come from the postings alone and a delete recomputes the exact keys from the old text; - one statistics entry [0x02][documents][terms], kept in the same transaction as the postings. Text is analyzed with TextNormalizer (accents removed, lowercased, punctuation as separators) and split into terms of at most 32 UTF-8 bytes. API: - HasFullTextIndex(x => x.Text) and CreateFullTextIndexAsync(x => x.Text) (default name idx_fts_{property}), filled from existing documents before the index is registered; - LINQ x.Text.FullTextMatch("terms"): documents containing every term, served from the index and evaluated with the same rules in memory; - FullTextSearchAsync(index, query, k, matchAllTerms) returning the top k documents with their BM25 score; - DynamicCollection.CreateFullTextIndexAsync / FullTextSearchAsync. Equality, range and IN lookups no longer pick a vector, spatial or full-text index just because it covers the property. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(fts): address review of full-text search (#70) - Tag postings 0x10 and statistics 0x20, outside the range of the generic [0x00..0x03] keys an older BLite writes when it reads the tree as a plain B-Tree; validate the statistics key length. - Record FullTextIndex.FormatVersion as the index KeyFormat and rebuild the index on open (typed and dynamic) when it differs or the tree holds keys the index did not write, since deletes rely on reproducing the keys. - Re-check every hit against the document text, for LINQ FullTextMatch and FullTextSearchAsync (typed and dynamic), so a stale posting never surfaces the document that now holds its location. - Search with matchAllTerms intersects rarest-first before scoring, and falls back to statistics estimated from the postings when the entry is missing (e.g. lost to a concurrent transaction's update). - CreateFullTextIndexAsync throws when the name belongs to a non-full-text index, passes its token to the rebuild, and is declared on IDocumentCollection; DynamicCollection.FullTextSearchAsync takes a token. - Document the concurrent-transaction statistics caveat, batching inserts and the CJK / supplementary-plane limitation. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(fts): batching advice and name-collision exception on CreateFullTextIndexAsync (#70) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * test(fts): a failed rebuild triggered by foreign keys is retried on the next open (#70) A full-text index whose format is current but whose tree holds keys it did not write (e.g. from an older BLite reading it as a plain B-Tree) is rebuilt on open. MigrateLegacyDecimalIndexes (on #175's base) rebuilds it into an unregistered copy and swaps that in only after it committed, so a rebuild that fails leaves the old index registered as it was: still holding the foreign keys, still needing the migration. - MigrateLegacyDecimalIndexes doc comment covers full-text indexes. - Test: corrupt a document so the rebuild on open fails; over three failing opens the stored root and format are unchanged, the foreign key is still there and the file does not grow; after repairing the document the next open swaps in the rebuilt index and a further open keeps it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(fts): return k re-checked hits and each location once (#70) - FullTextSearchAsync (typed and dynamic) keeps taking candidates past the first k, in widening windows, until k hits survive the re-check against the document; the first window costs one Search as before. - Postings yields one posting per location, so a stale posting sharing its term and location with the current document's no longer makes Match return the location twice or Search score it twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(fts): rarest-first saves decoding and allocation, not I/O (#70) CountRange walks every leaf entry of each query term, so looking up the few candidates in a common term's postings avoids decoding them, not reading their pages. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(fts): a stale posting at a document's location can be the one scored (#70) - Postings documents that the first posting of a location in key order (lowest frequency, then shortest length) is returned, which can be a stale one, so its frequency and length can be used for BM25 scoring, and CountRange counts both towards the document frequency. Matches and once-per-location results are unaffected. - The test for a stale posting at a document's location notes that it checks membership and order, not scores. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * fix(fts): free a failed dynamic full-text rebuild and keep the old index (#70) DynamicCollection rebuilt a stale full-text index on open into a fresh tree and, if a document could not be read or indexed, rolled back and threw without freeing that tree. Pages added by splits are allocated outside the transaction, so every failing open leaked the partial tree and grew the file. CreateFullTextIndexAsync leaked the same way. - FillUnregisteredFullTextIndexAsync fills the new index in its own transaction; on failure it collects the index's pages through the transaction before the rollback and frees them. Used by the rebuild on open and by CreateFullTextIndexAsync. - PublishFullTextIndex registers the filled index and persists the metadata; if that fails it restores the previous registration and frees the new index. - Each full-text index keeps its own KeyFormat and PersistIndexMetadata writes it, so persisting after one index's rebuild no longer marks another index that still awaits its rebuild as current (e.g. a crash between two rebuilds). Test: a dynamic full-text index over 3000 documents with a foreign key and an unreadable last document; four failing opens do not grow the file and keep the old index (same root, still searchable, still flagged for rebuild), and the rebuild succeeds once the document is repaired. Fails without this change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * feat(fts): record full-text indexes as a required feature of the collection (#70) Uses the per-collection required features from #194, so BLite versions that have that check but not full-text indexes refuse the collection instead of misreading it. - CollectionFeatures.FullTextIndexes: id "fulltext-index", description "full-text indexes (BLite 5.3 or later)", requirement Open. A version without full-text indexes reads the index as a B-Tree: its planner answers equality, range and IN lookups on the property from the postings, so even read-only results are wrong. It is a known feature in this version. - SyncIndexFeatures records the feature while the collection has a full-text index and removes it with the last one, leaving every other feature alone. Called whenever the index list is written: typed UpdateMetadata (CreateFullTextIndexAsync, model-declared indexes via ApplyIndexBuilder, rebuilds, DropIndex) and dynamic PersistIndexMetadata. - A read-write open records the feature again if a release that predates it rewrote the record and dropped the section (5.2.x does); a read-only open leaves the file alone. - CollectionFeatures and ReadCollectionMetadata document that the index-type rule only helps while an old writer keeps the index entry (a 5.2.x dynamic collection drops both), and that new per-index data goes only in trailing sections. - CreateFullTextIndexAsync (typed and dynamic) notes: do not open a database with full-text indexes for writing with BLite < 5.3. Tests: the feature is recorded for model-declared, typed and dynamic indexes, survives reopen, stays while another full-text index is left and is cleared with the last one; it is recorded again on a read-write open after being dropped, not on a read-only one; a version without full-text indexes (known features and index types passed in) refuses to open the collection even read-only, from the feature alone and from the index type alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(storage): drop the format-rule note duplicated by #219's remarks Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> Co-authored-by: MrDevRobot <mrdevrobot@entgldb.com>
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to the #194 review (#187).
Changes
SaveCollectionMetadataandDeleteCollectionMetadatanow throwNotSupportedExceptionwhen the collection's stored record cannot be parsed. The message names the collection and this version, and the record is left unchanged. Before,TryReadRecordswallowed every exception, so the feature check was skipped and the record was overwritten. Such a record also makesGetCollectionMetadatareturn null, so a dynamic open saved a fresh, empty record over it.EnsureCanReplaceRecordreplacesTryReadRecordin the same place: right afterFindMetadataRecord, before the in-place step.LoadMetadataChain/FindMetadataRecordinstead ofGetCollectionMetadata, which skips records it cannot parse.CollectionFeaturesremarks). The rule only protects a new index type if the older release that rewrote the record kept the index entry. Typed collections and saves that start from the stored record keep it. 5.2.3'sDynamicCollection.PersistIndexMetadatadoesn't: it rebuilds the index list from the indexes it loaded, on every dynamic index create/drop and on a vacuum withRebuildIndexes, and so drops the entry together with the feature section.CollectionFeaturesandReadCollectionMetadata): new per-index data goes only in trailing sections, never inline in the index list. An older version reads the list by position. Unexpected bytes there make the record fail to parse;GetCollectionMetadatathen returns null, a dynamic open saves a fresh record, and the feature refusal never runs.<Version>, andnuget-publish.ymlpasses-p:Version=<tag>. A build with-p:Version=5.3.0-beta.1has assembly version 5.3.0.0, so the message says "BLite 5.3.0".Tests
RecordThisVersionCannotParse_IsNeitherOverwrittenNorDropped: a record is cut right after its name.SaveCollectionMetadataandDeleteCollectionMetadatathrow.GetOrCreateCollectionthrows.The test fails with
main's guard, and also with only the Delete guard reverted.RequiredFeaturesTests+MetadataChainTests: 18/18.BLite.Corebuilds for net10.0 and netstandard2.1 with no warnings. Full suite: 2519 passed; the 5 failures were lock timeouts under heavy parallel load, and each passed when rerun on its own.BLite.NetStandard21.Testscouldn't run locally (it needs the net6.0 runtime).Release notes (suggestion)
🤖 Generated with Claude Code