Repository navigation
Data loss: with an encrypted WAL in multi-process mode, processes rewrite the WAL crypto header and erase or garble each other's commits #203
Description
Data loss. Reproduced on main at 870c729, with two StorageEngine instances standing in for two processes. Found while working on #198/#202.
With an encrypted WAL (PageFileConfig.CryptoProvider) and AllowMultiProcessAccess, each process manages the WAL's crypto file header on its own. The header carries a fresh random salt, and the WAL key is derived from it (AesGcmCryptoProvider.GetFileHeader). WriteAheadLog writes a header whenever its _cryptoInitialized flag is false, which happens when the WAL was empty at open or after this process truncated it. To do that, EnsureCryptoHeader first sets the file length to 0. Two things go wrong:
- A process that writes a header wipes other processes' commits. If the WAL holds another process's records when this process writes its header,
SetLength(0)erases them. - Processes end up with different keys. After a truncation, the process that writes the next header derives a new key. Processes that loaded the earlier header keep encrypting and decrypting with the old key. Neither side can read the other's records: decryption fails and the replay stops.
Repro 1: both processes open while the WAL is empty
Pages p and q are initialised to 0x01 and checkpointed by a third engine, so the WAL starts empty.
| Step | Result |
|---|---|
A commits p = 0xAA |
WAL 16,595 bytes (A's header + record) |
B commits q = 0xBB |
WAL still 16,595 bytes: B wrote a new header over A's commit |
| A begins, reads q | 0x01 (expected 0xBB) |
| B begins, reads p | 0x01 (expected 0xAA) |
| Reopen, read p and q | p = 0x01 (A's acknowledged commit is lost), q = 0xBB |
Repro 2: after a truncation
A commits, B opens after A wrote the header (as EncryptedWal_CrossEngine_CommitVisibleViaPhase7Replay does), and B replays A's commit correctly. Then:
| Step | Result |
|---|---|
| A checkpoints and truncates | WAL 0 bytes |
B commits q = 0xBB |
B still uses the old key |
A commits r = 0xCC |
A writes a new header and key |
| B begins, reads r | 0x01 (expected 0xCC) |
| A begins, reads q | 0x01 (expected 0xBB) |
| Reopen, read q and r | q = 0xBB, r = 0x01 (A's acknowledged commit is lost) |
The existing encrypted multi-process test passes only because B opens after A has already written a header and nothing truncates the WAL.
Possible fix
- Write the header once per shared WAL, under coordination. In multi-process mode, create the header only under the SHM writer lock, and only if the file is still empty after taking the lock. Otherwise load the header that is there. Never
SetLength(0)a WAL that may hold other processes' records. - Keep the header across truncations (truncate to
FileHeaderSize, not 0), so the key stays the same for the life of the WAL file and every attached process can keep using it. - If the salt really has to change per WAL generation: a process that sees a new generation (the counter from fix(storage): discard pages from a WAL another process has truncated (#197) #201) must reload the header before reading or appending.
Related: #201 and #202 fall back to the old behaviour for an encrypted WAL whose published end lies inside the header, so they neither fix nor worsen this.
Activity