Multi-process: RecoverAsync on open truncates the shared WAL without the SHM lock and leaves the SHM end offset stale #200

Found by @mrdevrobot while reviewing #169; plausible, not reproduced yet. The code references are to the #169 branch (b721fee).

Opening a database whose WAL is not empty runs StorageEngine.RecoverAsync (StorageEngine.Recovery.cs ~L329-383). It replays the WAL and then truncates it. In multi-process mode this truncation:

• doesn't take the SHM writer lock, so it can race with another process that is appending;
• doesn't reset the WAL end offset kept in the SHM.

WalSharedMemory.AdvanceWalEndOffset only ever moves the offset forward. So after this truncation, the processes that are already running keep a WAL end that is larger than anything newly appended. They don't see new records until the end passes that old value, and the checks that compare against it are off in the meantime.

Possible fix

In multi-process mode, run recovery's truncation under the SHM writer lock and reset the SHM end offset, the same way the checkpoint path does. Or skip the truncation on open when other processes are attached and leave it to the next checkpoint.