Multi-process WAL catch-up in BeginTransaction reads the WAL without the SHM writer lock #199

Found by @mrdevrobot while reviewing #169; plausible, not reproduced yet. The code references are to the #169 branch (b721fee).

The catch-up in StorageEngine.BeginTransaction (through TryCatchUpWithSharedWal) reads the shared WAL file without the SHM writer lock. If another process truncates the WAL and appends new records while the read is in progress, the replay can parse the new bytes at the old offsets. It can then misread records, or skip records and still move the bookmark forward.

Possible fix

A WAL generation counter in the SHM, incremented on every truncation. Read it before and after the replay, and throw the result away if it changed. This is the counter suggested in #197, and it would cover both problems.