fix(storage): discard pages from a WAL another process has truncated (#197)

[CaffeinatedCoder]

Fixes #197. Also covers #199.

In multi-process mode, committed data was lost once another process had checkpointed and truncated the shared WAL. Working on #197 turned up three more ways to the same loss (the third, through the backup path, is described under Changes), one of them much more common than the sequence in #197. A randomized two-engine stress run, described below, lost commits in 76 of 200 runs on #169's code and in none with this PR.

What went wrong

1. Data loss in multi-process mode: pages replayed before another process truncated the WAL are written back over newer data #197: pages from the old WAL. Pages an engine had replayed, or read through the SHM page index, stayed in its _walIndex after another process checkpointed and truncated the WAL. Its reads returned them, and its next checkpoint wrote them over the newer images on disk. This does not even need the second BeginTransaction from Data loss in multi-process mode: pages replayed before another process truncated the WAL are written back over newer data #197: CheckpointAsync never caught up before writing.
2. A WAL that grew past the old bookmark. Truncation was only detected as "SHM end < bookmark". Once the new WAL had grown past the old bookmark, the engine replayed it from that offset and skipped its first records. It then counted itself caught up, checkpointed, and truncated those records away.
3. Abort records written at a stale position. RollbackTransactionAsync appended an abort record outside the commit path: no SHM writer lock, and at the engine's own write position, which another process's appends or truncation can make stale. After a truncation that left a gap of zeros in the shared WAL, and the replay of every other process stopped at the gap (the "end of written area" check). They skipped the commits after it, counted themselves caught up, and truncated those commits. Every transaction disposed without a commit writes such a record, read-only ones included, so this was the most frequent case in the stress run. It isn't on the Multi-process WAL: PrepareTransactionAsync and the direct commit paths bypass the commit and cross-process writer locks #188 list.

Changes

• WAL generation in the SHM header.
  • Offset 48, in bytes that were reserved, so the version and size stay the same; existing files read 0. Every truncation increments it as its last step, under the writer lock, after resetting the end offset.
  • Processes running an older release don't increment it. Their truncations are still caught by the "end < bookmark" check, but not once the WAL has regrown.
• Catch-up (TryCatchUpWithSharedWal).
  • If the generation differs from the one the engine's state belongs to, or the end is before the bookmark, it drops everything it holds from the old WAL and replays the new one from its start. This is safe because the truncating process had checkpointed every record, including this engine's own commits: a commit replays the WAL, and so notices a truncation, before it appends.
  • Without the writer lock (BeginTransaction), a truncation during the replay discards the result. That covers Multi-process WAL catch-up in BeginTransaction reads the WAL without the SHM writer lock #199.
• Reads through the SHM page index. Such a page is cached in _walIndex only if neither the generation nor a local discard counter changed while it was read. Both are checked under a small lock, which only the read-from-WAL path takes. Otherwise a read that started before a discard could put an old image back.
• CheckpointAsync in multi-process mode.
  • It catches up and writes its pages under _commitLock and the SHM writer lock, in the same order as the commit path. So it writes the newest image of every page, and no other process can commit or truncate while it does.
  • Commits in all processes wait for the page writes; the flush runs after the locks are released.
  • Backup already holds _commitLock, so it uses the same path without taking that lock again.
  • Single-process mode is unchanged.
• Backup. In multi-process mode the backup truncated the WAL under _commitLock alone: no SHM writer lock, no reset of the shared end offset, and no check for records other processes hadn't replayed yet. That was another route to the Data loss in multi-process mode: pages replayed before another process truncated the WAL are written back over newer data #197 loss. It now truncates through the same path as a checkpoint, and so does the checkpoint the backup runs first.
• Abort records. In multi-process mode the abort record is no longer written. Replay and recovery ignore it, since only transactions with a commit record are applied.
• Docs.
  • README: the line you asked for in fix(storage): don't reuse freed pages that still have a WAL image (#168) #169 (BeginTransaction can wait up to ReadTimeoutMs), plus that a checkpoint makes commits wait for its page writes.
  • The SHM layout table in MULTI_PROCESS_WAL.md lists the new field.

Tests

MultiProcessWalTruncationTests, with two engines on the same files. All seven fail on b721fee (#169 as merged):

• the Data loss in multi-process mode: pages replayed before another process truncated the WAL are written back over newer data #197 sequence;
• the same without a second transaction in B, checkpointing straight away;
• a WAL that regrew past B's bookmark: B begins without reading, then checkpoints, and a fresh engine must still find the new WAL's first record;
• B's own commit after A truncated: it is kept, and A's newer page isn't overwritten;
• a truncation by A's backup: B then drops its replayed page;
• a rollback in A after B truncated, followed by a commit from A, which must survive B's checkpoint;
• 25 seeded random interleavings of 300 steps (commits, empty and rolled-back transactions, reads checked against the last commit, checkpoints from either engine). A fresh engine must read the last committed value of every page.

I disabled each part in turn and checked what fails:

Part disabled	Tests that fail
Generation comparison in the catch-up	the regrown-WAL test
Discarding the old WAL's pages	the #197 sequence, the variant without a second transaction, the own-commit test
Catching up before a checkpoint writes	the variant without a second transaction
Incrementing the generation on truncation	the regrown-WAL test
Skipping the abort record	the rollback test, the random interleavings
Backup and its checkpoint truncating through the coordinated path	the backup test

The read-through cache guard and the "truncated during an unlocked replay" check close races that need a truncation at an exact instruction, so no deterministic test covers them.

Stress run, 200 seeds × 300 steps, same harness as the last test:

Code	Runs that lost commits	Pages lost	Stale reads inside a transaction
#169 (b721fee)	76	106	528
b721fee plus only skipping the abort record	26	31	314
this PR	0	0	0

The multi-process, free-page, SHM, checkpoint, backup and rollback tests (91) passed repeated runs with no hangs. Full suite: 2409 passed, 0 failed, 4 skipped; on current main (with #169 and #170): 2417 passed, 0 failed, 4 skipped. BLite.NetStandard21.Tests: 20/20. BLite.Core builds with no warnings. The branch merges with main and every open PR without conflicts.

Not changed here

• Multi-process WAL catch-up: a replay that stops at a torn record still advances the bookmark past later commits #198: a torn record from a crashed writer still ends a replay early without an error. After this PR a rollback no longer creates such a gap, but a crash can.
• Multi-process WAL: PrepareTransactionAsync and the direct commit paths bypass the commit and cross-process writer locks #188: PrepareTransactionAsync and the direct commit paths still append without the SHM writer lock.
• Multi-process: RecoverAsync on open truncates the shared WAL without the SHM lock and leaves the SHM end offset stale #200: RecoverAsync on open still truncates without it.

🤖 Generated with Claude Code

[mrdevrobot]

Thanks. The generation counter in the reserved SHM bytes is a clean way to detect truncation without a version bump, and both old readers (ignore offset 48) and old SHM files (generation 0) are handled. Discarding all of _walIndex is safe for group commits, because truncation only happens when the truncating process has replayed and checkpointed everything published. #197 reproduces on main, and each part of the fix is covered by a test that fails when it's reverted. Lock order is consistent, BeginTransaction takes no new lock, and single-process mode is untouched. Full suite merged with main 2464 / 0 / 4 skipped. This also addresses #199.

One change: the discard turns commits from the direct paths into lost commits (StorageEngine.Transactions.cs:132, with CommitTransactionAsync(ulong) at :277 and MarkTransactionCommittedAsync at :451). Those paths don't catch up and don't publish the SHM end (#188), so "a commit replays the WAL before appending" only holds for group commit. Sequence: B commits P = 0x44 through CommitTransactionAsync(ulong); A commits Q, checkpoints and truncates without ever seeing P; B begins, discards, and reads the old P. The commit is gone. On main B kept P and its own checkpoint wrote it. I have a probe that passes on main and fails here. Please either land #188 first, or throw NotSupportedException from the direct and 2PC paths when AllowMultiProcessAccess is set.

Also:

• A failed catch-up silently skips the checkpoint (StorageEngine.Recovery.cs:80): nothing is written, not even this engine's own pages, there's no log, and the Checkpoint metric still reports success. Please report it as unsuccessful and emit a warning or metric.
• Mixed versions keep the old bugs: older processes don't bump the generation or discard, so #197 still happens in a mixed deployment. Please document in MULTI_PROCESS_WAL.md and the README that every process must run this version or later.
• InitializeHeader (page size or max-readers mismatch) resets the generation under live processes, which they read as a truncation. Configuration error only, but worth a guard.
• The backup's second TryTruncateWalCoreAsync call is redundant: with only it reverted, all 7 tests still pass.

[CaffeinatedCoder]

Thanks. Addressed in 829b471, after merging main in de840ae:

• Direct and 2PC paths: I reproduced your sequence on 590174d: B reads 0x11 instead of 0x44.
  • In multi-process mode, CommitTransactionAsync(ulong), both PrepareTransactionAsync overloads and MarkTransactionCommittedAsync now throw NotSupportedException before writing anything. The transaction stays usable and can still commit through group commit.
  • Nothing in the library calls these paths (Transaction.CommitAsync uses the (ulong, CancellationToken) group-commit overload), so I went with the gate rather than routing them. 2PC can't be routed without redefining what a prepare guarantees, which stays with Multi-process WAL: PrepareTransactionAsync and the direct commit paths bypass the commit and cross-process writer locks #188.
  • Tests: DirectCommitPath_IsRejected_SoAnotherProcessesTruncationCannotLoseIt (your sequence), TwoPhaseCommitPaths_AreRejected and DirectAndTwoPhaseCommitPaths_StillWork_InSingleProcessMode.
• Failed catch-up / checkpoint: a checkpoint that can't take its locks or catch up now publishes the Checkpoint metric with Success = false.
  • A new MetricsSnapshot.CheckpointsFailedTotal counts these; they no longer count in CheckpointsTotal or the average latency.
  • A failed catch-up is also traced as a warning (Trace.TraceWarning, as in the free-list fix): the first, then every power of two in a row.
  • Test: CheckpointThatCannotTakeTheWriterLock_IsReportedAsFailed.
• Mixed versions: README and MULTI_PROCESS_WAL.md now say every process must run a version later than 5.2.3, and explain why.
• InitializeHeader: a well-formed SHM with another page size or reader count is re-initialised only if it shows no sign of use (WAL end 0, no live writer or reader PID). Otherwise Open throws InvalidOperationException.
  • When it is re-initialised, the generation is incremented instead of reset, because a reset could later repeat a generation an attached process still holds. NextTransactionId is kept.
  • Known limit: an idle process attached to an empty WAL leaves no trace, so a reader-count mismatch with one can't be detected. StorageEngine always uses the default count.
  • Tests: ShmOpenedWithAnotherPageSize_WhileInUse_ThrowsAndKeepsTheHeader, ..._WhenIdle_IsReinitialisedWithoutGoingBack.
• Backup: the second TryTruncateWalCoreAsync call is removed. The only case it covered (empty _walIndex, non-empty WAL) is safe, because backup copies the WAL.

Each new test fails with its part reverted. The multi-process filter passed 5 runs in a row with no hangs. Full suite: 2466 passed / 4 failed / 4 skipped. The 4 failures are lock-timeout TimeoutExceptions in single-process tests under parallel load, and they pass in isolation, except CrossCollectionWriteRaceTests.Concurrent_Inserts_Into_Different_Collections_LoseNothing, which is also flaky on main here (2 of 6 runs).