fix(storage): don't let lock-free metadata readers miss records a compaction moves (follow-up to #196)

[CaffeinatedCoder]

Follow-up to #196, stacked on fix/195-metadata-growth; review #196 first. Until #196 merges, the diff here also shows #196's commits; only 445d683 is new. Refs #195.

This addresses the follow-up requested in the #196 review.

Lock-free readers vs. compaction

• GetCollectionMetadata keeps its lock-free walk, and a record it finds is returned as before, since saves only rewrite or move records. A miss, or a page that cannot be parsed, is now repeated under _metadataLock. Before, a compaction could move collection B onto a page the reader had already read, and rewrite or free the page B was on. The reader then returned null, and the DynamicCollection / CollectionIndexManager constructors saved fresh metadata over B's record.
• GetAllCollectionsMetadata now walks under _metadataLock. For a full listing a miss can't be detected afterwards: records moved behind the reader leave neither a null nor a duplicate. Its callers (ListCollections, DropOrphanCollections, GDPR inspection) are not hot paths.

Compaction cost

• CompactMetadataChain(onlyIfShorter: true) returns immediately for a one-page chain.
• For longer chains it plans the packing from slot lengths and copies nothing unless it actually rewrites. The rewrite appends straight from the loaded page buffers.
• Documented that a crash between the last compaction write and the FreePage loop leaks the tail pages (file space only).

Tests (MetadataChainTests)

• The race, made deterministic with an IPageStorage wrapper that runs the compaction on another thread right after the reader reads a given page.
  • Two shapes: the reader then sees page 2 already rewritten, or it follows the old link into a freed page.
  • Covered for GetCollectionMetadata (both shapes) and GetAllCollectionsMetadata.
  • All 3 tests fail on fix(storage): stop the collection metadata chain from growing with every save (#195) #196 as it stands: null, or a listing missing 12 collections.
• Up-front size check: a record of exactly the page maximum is stored, and one byte more throws InvalidOperationException and leaves the chain and the stored record unchanged. With the check removed, the oversized record is accepted silently: its slot entry overlaps the record's first byte, and the old record is marked deleted.

Timings (Release, file-backed, median µs)

	#196	this PR
GetNextSequence, 1-page chain	66	60–72
GetNextSequence, 3-page chain	95	75–82
GetCollectionMetadata hit	1.3	1.2–1.4
GetCollectionMetadata miss (now a second, locked walk)	1.3	2.6

Full suite: 2402 / 0 / 4 skipped (branch not yet merged with main).

Note: a miss in GetCollectionMetadata, and every GetAllCollectionsMetadata call, now wait for _metadataLock under the same WriteTimeoutMs as saves.

🤖 Generated with Claude Code

[CaffeinatedCoder]

Updated to main (merge 0a42bef; #196 and #194 are in), so the diff now shows only the follow-up: StorageEngine.Collections.cs and MetadataChainTests.cs.

• feat(storage): per-collection required features, refused by versions that lack them (#187) #194's guard placement and read-path checks are kept. The feature checks run on the metadata the readers return, and the locked re-walk uses the same parser, so they apply there too.
• One merge-specific change: DeleteCollectionMetadata (feat(storage): per-collection required features, refused by versions that lack them (#187) #194's guard) looked the record up through GetCollectionMetadata while holding _metadataLock. With the locked re-walk, deleting a missing collection would have waited on that non-reentrant lock until it timed out. It now uses FindCollectionMetadata. New test DeleteOfAMissingCollection_DoesNotWaitForTheLockItHolds fails with the old call.
• On the merged code, the race tests still fail with the reader fix reverted. The full suite passes except for timing-sensitive concurrency tests under parallel load. One of them, HighContention_ManySmallTransactions_ConsistentFinalState, fails the same way on main.

[mrdevrobot]

Thanks. Retrying a lock-free miss under _metadataLock keeps the hot hit path lock-free and closes the case where a compaction moved a record behind the reader and the constructors saved fresh metadata over it. The deterministic race tests with the IPageStorage wrapper are very good. Checked that no caller holding _metadataLock reaches GetCollectionMetadata/GetAllCollectionsMetadata (GetNextSequence uses its own per-collection lock).

I merged main into the branch (932a35d) to resolve the conflict with #219 in DeleteCollectionMetadata: kept EnsureCanReplaceStoredRecord, which reads the chain directly and so doesn't re-enter the lock. Full suite on the result: 2538 passed, 0 failed.