Multi-process WAL: a permanently unreadable record inside the published range stalls replay, checkpoints and truncation #220

Found during the review of #202.

A process catching up with the shared WAL replays [bookmark, SHM WalEndOffset). Since #202, a record it can't read (torn, corrupt, or past the end of the file) ends the catch-up as incomplete, and the bookmark stays put. That avoids skipping or truncating later records (#198). But if the record stays unreadable:

• every catch-up (BeginTransaction, group commit, checkpoint) fails for every process whose bookmark is before it;
• those processes' checkpoints write nothing, not even their own pages, and they never truncate the WAL;
• their _walIndex and the WAL grow without bound, and their reads may miss other processes' commits.

The stall only ends if the record becomes readable again, or if a process whose bookmark is already past the record (e.g. the one that wrote it) checkpoints and truncates the WAL.

Causes:

• disk corruption inside the published range;
• a record written at a stale offset by a path that bypasses the locked append sequence (Multi-process WAL: PrepareTransactionAsync and the direct commit paths bypass the commit and cross-process writer locks #188; these paths now throw in multi-process mode);
• Multi-process: RecoverAsync on open truncates the shared WAL without the SHM lock and leaves the SHM end offset stale #200: after the unlocked RecoverAsync truncation on open, the SHM WalEndOffset stays past the file's end, because AdvanceWalEndOffset only moves forward. Every replay is then incomplete until the WAL grows past the old end. This is a stall, not a loss.

Since #202 the stall is visible. A trace warning gives the WAL path and the offset: the first time, then every power of two in a row, and from 16 in a row it says "stalled". Failed checkpoints are counted in MetricsSnapshot.CheckpointsFailedTotal. Nothing recovers from it automatically.

Possible directions:

• Fix Multi-process: RecoverAsync on open truncates the shared WAL without the SHM lock and leaves the SHM end offset stale #200: truncate under the SHM writer lock and reset the end offset, or skip the truncation on open while other processes are attached.
• After a stall threshold, let a process that holds the writer lock and has checkpointed everything it can read trigger a recovery: replay the WAL up to the bad record, checkpoint, truncate, start a new generation. This deliberately gives up the records past the bad one, so it needs a decision on whether that's acceptable or whether to stop with an error.
• Expose the stall as a metric (e.g. a consecutive-failed-catch-up gauge), not only as a trace warning.