fix(storage): don't skip WAL records past a torn one in multi-process replay (#198)

[CaffeinatedCoder]

Fixes #198.

Stacked on #201. Only the last commit (fix(storage): don't skip WAL records past a torn one in multi-process replay) belongs here. Both change TryCatchUpWithSharedWal, so basing this on main would conflict with #201. I'll merge main in once #201 lands.

Problem

WriteAheadLog.ReadCommittedPagesSince stops, without an error, at a record it can't read: a short read, an implausible size, a decryption failure, an unknown type or a zero byte. TryCatchUpWithSharedWal still reported success and moved the bookmark to the SHM end. So the engine skipped every record after the unreadable one, which could include other processes' commits. Its next checkpoint could then truncate the WAL that still held them, and with #201's discard on truncation those commits are gone for good.

How a torn record gets into the middle of the WAL: a writer dies part-way through a batch. Its bytes stay in the file beyond the published WAL end, and the next writer appended after them, because SeekToEnd went to the file's end. Every replay that started before the torn bytes then stopped there.

Changes

• Reader: ReadCommittedPagesSince reports whether its records ended exactly at the requested end. If they didn't (it stopped early or ran past the end), TryCatchUpWithSharedWal returns false and the bookmark stays. The engine then neither counts the records as replayed nor truncates the WAL, until a replay gets through.
• Writer:
  • The group commit appends at the published SHM WAL end and cuts off anything beyond it (SeekToSharedEnd, under the writer lock).
  • Those bytes belong to a batch that was never published. A commit is acknowledged only after the SHM end has moved past it and the locks are released, so nothing acknowledged is dropped.
  • Without the cut-off, the torn bytes would block every replay that reaches them, and no process could truncate the WAL again.
  • It falls back to the end of the file, as before, when the published end lies beyond it, or inside an encrypted WAL's crypto header.

Tests

Two new tests in MultiProcessWalTruncationTests. Both fail on the parent commit (#201), and each fails when its own half of the fix is removed:

• TornBatchOfACrashedWriter_IsCutOff_SoTheNextCommitIsReplayed: unpublished partial bytes of a write record are appended to the WAL, then A commits. B must replay that commit, checkpoint and truncate the WAL, and every page must survive. Without the cut-off, B's replay stops at the torn bytes and B can't truncate.
• UnreadableRecordInsideTheWal_StopsTheReplay_WithoutLettingTheReaderTruncate: the first of A's records is corrupted. B commits and checkpoints its own page, and the WAL must stay as it is, so A's pages and B's page all survive A's checkpoint. Without the check, B truncates the WAL and A's later records are lost.

The full suite passes (2419 passed, 0 failed, 4 skipped), and so do BLite.NetStandard21.Tests (20/20). The 196 WAL, multi-process and encryption tests passed three runs in a row. BLite.Core builds with no warnings.

Limits

• Corruption inside the WAL, not a crash: a record that is unreadable for any other reason (a disk error, say) still stops every replay that reaches it. Readers then stay behind and no process truncates the WAL, so it keeps growing. That is the safe failure mode, instead of silently skipping commits. What recovery on the next open does with such a record is unchanged.
• Encrypted multi-process WAL: each new WAL gets a crypto header with a fresh random salt. After a truncation, processes that loaded the old header encrypt with a different key than the one now in the file. That predates this PR and isn't changed here; it's Data loss: with an encrypted WAL in multi-process mode, processes rewrite the WAL crypto header and erase or garble each other's commits #203, which also shows the WAL being emptied under another process's commits.
• Multi-process WAL: PrepareTransactionAsync and the direct commit paths bypass the commit and cross-process writer locks #188 and Multi-process: RecoverAsync on open truncates the shared WAL without the SHM lock and leaves the SHM end offset stale #200 are unchanged. PrepareTransactionAsync and the direct commit paths still write without the writer lock. Recovery on open still truncates without it.

🤖 Generated with Claude Code

[mrdevrobot]

Reviewed only f38949d. The reader side is right: ReadCommittedPagesSince now reports whether it reached toOffset, and on an incomplete scan the bookmark stays put. #198 reproduces on main, and both new tests fail with their part reverted. Filtered multi-process tests passed 5 runs in a row with no hangs; full suite merged with main 2466 / 0 / 4 skipped.

Two things:

1. The cut-off deletes acknowledged commits (WriteAheadLog.cs:486-502, SeekToSharedEnd / SetLength(sharedEnd), called from StorageEngine.GroupCommit.cs:171). It removes everything past the SHM end, which includes records written by CommitTransactionAsync(ulong) and MarkTransactionCommittedAsync (they never publish the end) and prepared records from PrepareTransactionAsync. "A commit is acknowledged only after the SHM end has moved past it" doesn't hold for those paths. Repro with a single engine and AllowMultiProcessAccess: commit P = 0x44 through the direct path, commit Q through group commit, close, reopen: P reads as 0. It passes on main and #201 and fails here. With PrepareTransactionAsync + MarkTransactionCommittedAsync, the commit record can survive while its write records are cut. Same fix as on #201: gate those paths in multi-process mode or route them through the locked append-and-publish sequence (#188).

2. A permanently unreadable record inside the published range stalls the WAL. Torn bytes from a crashed writer sit past the SHM end, so the cut-off handles them. A bad record before the end (disk corruption, or a #188 direct write at a stale offset) makes every process whose bookmark is behind it fail every catch-up, skip every checkpoint and never truncate; the WAL and _walIndex grow without bound and nothing reports it. You note the limit; please surface it through a log or metric and file an issue. Related: after #200's unlocked RecoverAsync truncation, the SHM end stays past the file end, so every replay is incomplete until the WAL grows past it (a stall, not a loss).

This needs to rebase onto main after #201 lands.

[CaffeinatedCoder]

Thanks. #201's new head is merged in (dae4638); changes in 9596837:

1. Cut-off vs. direct/2PC commits: I confirmed your repro (direct P, group Q, close, reopen → P reads 0).
   • With fix(storage): discard pages from a WAL another process has truncated (#197) #201's gate, CommitTransactionAsync(ulong), PrepareTransactionAsync and MarkTransactionCommittedAsync throw in multi-process mode before writing, and the abort record is already skipped there.
   • That leaves group commit as the only multi-process WAL writer, and it publishes everything it acknowledges. So the cut-off can only remove a batch that was never published. The one other unlocked writer is RecoverAsync's truncation (Multi-process: RecoverAsync on open truncates the shared WAL without the SHM lock and leaves the SHM end offset stale #200).
   • Test: CutOff_CannotDeleteCommitsOfTheDirectOrTwoPhasePaths (your repro; it also checks that the WAL length equals the SHM end before and after).
2. Stall: ReadCommittedPagesSince also returns where it stopped, and an incomplete catch-up is now reported like a failed one.
   • It emits a trace warning with the WAL path and the offset of the unreadable record: the first time, then at every power of two in a row. From 16 in a row the warning says the replay has stalled.
   • Checkpoints during the stall count in MetricsSnapshot.CheckpointsFailedTotal (from fix(storage): discard pages from a WAL another process has truncated (#197) #201).
   • The generation check now runs before the completeness check, so a truncation racing an unlocked replay isn't reported.
   • Documented in MULTI_PROCESS_WAL.md. Test: UnreadableRecordInsideThePublishedRange_IsReportedAsAStall. Issue for the stall and the Multi-process: RecoverAsync on open truncates the shared WAL without the SHM lock and leaves the SHM end offset stale #200 case: Multi-process WAL: a permanently unreadable record inside the published range stalls replay, checkpoints and truncation #220.

Both tests fail with their part reverted. The multi-process filter passed 11 runs with no hangs. Full suite: 2471 passed / 3 failed / 4 skipped; the 3 failures are lock timeouts under parallel load and pass in isolation. I'll rebase onto main once #201 lands.