Skip to content

Multi-process WAL catch-up: a replay that stops at a torn record still advances the bookmark past later commits #198

Closed

Description

@CaffeinatedCoder

Found by @mrdevrobot while reviewing #169. The defect is on main too; the code references are to the #169 branch (b721fee).

WriteAheadLog.ReadCommittedPagesSince(fromOffset, toOffset) (~L718-830) stops without an error when it hits a record it can't read: a short read, an implausible size, a decryption failure or an unknown type. Its caller, StorageEngine.TryCatchUpWithSharedWal, then still reports success and moves the bookmark to shmEnd.

So a torn record left by a crashed writer makes this engine skip every record after it. That includes later, valid commits from other processes, which this engine then never replays. Its next checkpoint can truncate the WAL that still holds them. Reproduced with a hand-truncated record.

Possible fix

Have the parser report whether it reached toOffset, for example by returning the offset it stopped at. TryCatchUpWithSharedWal should treat a short read as failure: leave the bookmark where it is and don't truncate the WAL. A torn record at the very end, written by a crashed process, needs a rule of its own. It can be skipped only once the writer lock shows that no process is still appending it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    • Status
      Done

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions