Skip to content

fix(archiver): clean up removed blocks from raw rows and ownership-check tx-effect deletes - #24765

Merged
PhilWindle merged 3 commits into
merge-train/spartan-v5from
spl/archiver-tx-effect-cleanup
Jul 27, 2026
Merged

fix(archiver): clean up removed blocks from raw rows and ownership-check tx-effect deletes#24765
PhilWindle merged 3 commits into
merge-train/spartan-v5from
spl/archiver-tx-effect-cleanup

Conversation

@spalladino

Copy link
Copy Markdown
Contributor

Problem

Two defects in BlockStore block removal could corrupt the tx-effect index (txHash → owning block hash/position) if the store ever held two blocks sharing a tx:

  1. Blind deletedeleteBlock removed #txEffects entries by txHash without checking the entry still pointed at the block being deleted. If another stored block contained the same tx, that block's entry was destroyed collaterally, making the block unreadable (Could not find tx effect for tx…).
  2. Skip-on-unreadable leakremoveBlocksAfter did getBlock(bn) === undefined → warn + continue, so an unreadable block's row, tx effects, and indices were never released. A later addCheckpoints overwrites the leaked row in place ("L1 data is authoritative"), leaving stale tx-effect entries pointing at coordinates now occupied by different txs — getTxReceipt then reports positions the chain no longer has, and getL2ToL1MembershipWitness resolves the wrong tx and throws The L2ToL1Message you are trying to prove inclusion of does not exist.

This is hardening, not an incident fix. No honest code path produces two stored blocks sharing a tx: duplicate nullifiers are rejected at block building (double-spend validation against the build fork), at validator re-execution, and at world-state sync (nullifier leaves are non-updateable). The incident that prompted the investigation turned out to be a downstream client bug following a v5 behavior change in L2BlockSynchronizer, unrelated to the store. The store should still not amplify a duplicated-tx state into silent corruption if it ever appears — e.g. via a future upstream bug, or byzantine tx effects that repeat a txHash with distinct nullifiers (invisible to every nullifier-based check).

Fix

  • removeBlocksAfter cleans up from the raw storage row, so blocks whose bodies can no longer be fully loaded still release their row, tx effects (enumerated from #blockTxs), and hash/archive indices.
  • deleteBlock only deletes a tx-effect entry still owned by the block being removed (compares the entry's leading block hash).
  • Both anomalies now log a warning (an entry owned by another block, or a missing entry). These states should be unreachable, so a warning in production logs is direct evidence of an upstream bug worth investigating — previously the ownership skip was silent, hiding exactly that signal.

Cost

The added reads are confined to prune/reorg paths: one #blockTxs read plus one tx-effect point read per tx of each removed block (roughly doubling point reads on block removal). Block ingestion and query paths are untouched.

Verification

Recreates #24732 with the original commit cherry-picked and authorship preserved (thanks @benesjan). On top of it, this PR updates the description and code comments to match what the follow-up investigation established about how a duplicated-tx state can and cannot arise, and adds the anomaly warnings. Supersedes #24732.

Related to A-1426

benesjan and others added 3 commits July 17, 2026 14:34
…ed elsewhere

BlockStore.removeBlocksAfter had two defects that corrupt the tx-effect
index (txHash -> block position) when the same tx exists in two stored
blocks, e.g. re-included after its original proposal expired:

- deleteBlock removed #txEffects entries blindly by txHash, destroying
  the entry of a tx whose index already points at another stored block,
  which makes that block unreadable.
- removeBlocksAfter skipped cleanup entirely for blocks it could not
  reconstruct, leaking their row, tx effects, and indices; a later
  insert at the same number then overwrites the row in place, leaving
  stale tx-effect entries pointing into the new chain.

A stale entry makes getL2ToL1MembershipWitness resolve the tx at wrong
coordinates and throw 'The L2ToL1Message you are trying to prove
inclusion of does not exist' for a message in a proven block, and lets
getTxReceipt report a proven position the chain no longer has.

Cleanup now works from the raw storage row (so unreadable blocks are
still fully released) and only deletes tx-effect entries still owned by
the block being removed.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…oval

The ownership-checked delete silently skipped entries owned by another
block. That state (two stored blocks sharing a tx) should be unreachable
for honest chains, so skipping it silently hides direct evidence of an
upstream bug. Warn on both anomalies (foreign-owned entry and missing
entry) and correct the comment that attributed the duplication to
routine proposal expiry, which cannot produce it.
@spalladino
spalladino force-pushed the spl/archiver-tx-effect-cleanup branch from 5a61d75 to 1f57f94 Compare July 17, 2026 17:34
@spalladino
spalladino changed the base branch from merge-train/spartan to merge-train/spartan-v5 July 17, 2026 17:35
PhilWindle added a commit that referenced this pull request Jul 27, 2026
…24754)

`getL2ToL1MembershipWitness` assembled its witness from several
non-atomic archiver reads (`getTxEffect`, then the epoch blocks, target
block and checkpoint metadata read inside
`computeL2ToL1MembershipWitness`). A store commit landing between those
reads could splice together two different chain states and surface a
spurious "message does not exist" error even when the tx-effect index
was healthy.

Wrap the witness assembly in a single `store.transactionAsync` so every
archiver read binds to one write transaction and observes a consistent
snapshot (the store serializes writers, so no commit can interleave).
The L1 Outbox roots fetch stays outside the transaction to avoid holding
the archiver's writer lock across a network round-trip, and the tx
effect is re-read inside the snapshot so the receipt's block number and
tx index stay consistent with the block data.

Cost: witness assembly briefly serializes with archiver writers (it
holds the store's writer queue while it reads), but the work under the
lock is pure store reads plus an in-memory tree build — no network I/O.

Context: this came out of the A-1426 investigation. The incident symptom
itself turned out to be a downstream client bug following a v5 behavior
change in `L2BlockSynchronizer` — not this race, and not archiver index
corruption. This PR is therefore hardening against a real but
so-far-unobserved torn-read window, not the incident fix. See #24765 for
the companion store hardening from the same investigation.

Related to A-1426
AztecBot pushed a commit that referenced this pull request Jul 27, 2026
…24754)

`getL2ToL1MembershipWitness` assembled its witness from several
non-atomic archiver reads (`getTxEffect`, then the epoch blocks, target
block and checkpoint metadata read inside
`computeL2ToL1MembershipWitness`). A store commit landing between those
reads could splice together two different chain states and surface a
spurious "message does not exist" error even when the tx-effect index
was healthy.

Wrap the witness assembly in a single `store.transactionAsync` so every
archiver read binds to one write transaction and observes a consistent
snapshot (the store serializes writers, so no commit can interleave).
The L1 Outbox roots fetch stays outside the transaction to avoid holding
the archiver's writer lock across a network round-trip, and the tx
effect is re-read inside the snapshot so the receipt's block number and
tx index stay consistent with the block data.

Cost: witness assembly briefly serializes with archiver writers (it
holds the store's writer queue while it reads), but the work under the
lock is pure store reads plus an in-memory tree build — no network I/O.

Context: this came out of the A-1426 investigation. The incident symptom
itself turned out to be a downstream client bug following a v5 behavior
change in `L2BlockSynchronizer` — not this race, and not archiver index
corruption. This PR is therefore hardening against a real but
so-far-unobserved torn-read window, not the incident fix. See #24765 for
the companion store hardening from the same investigation.

Related to A-1426
@PhilWindle PhilWindle added the port-to-next Forward-port this merged PR into next label Jul 27, 2026
@PhilWindle
PhilWindle merged commit 38bc4c1 into merge-train/spartan-v5 Jul 27, 2026
28 checks passed
@PhilWindle
PhilWindle deleted the spl/archiver-tx-effect-cleanup branch July 27, 2026 10:24
AztecBot pushed a commit that referenced this pull request Jul 27, 2026
…eck tx-effect deletes (#24765)

## Problem

Two defects in `BlockStore` block removal could corrupt the tx-effect
index (txHash → owning block hash/position) if the store ever held two
blocks sharing a tx:

1. **Blind delete** — `deleteBlock` removed `#txEffects` entries by
txHash without checking the entry still pointed at the block being
deleted. If another stored block contained the same tx, that block's
entry was destroyed collaterally, making the block unreadable (`Could
not find tx effect for tx…`).
2. **Skip-on-unreadable leak** — `removeBlocksAfter` did `getBlock(bn)
=== undefined → warn + continue`, so an unreadable block's row, tx
effects, and indices were never released. A later `addCheckpoints`
overwrites the leaked row in place ("L1 data is authoritative"), leaving
stale tx-effect entries pointing at coordinates now occupied by
different txs — `getTxReceipt` then reports positions the chain no
longer has, and `getL2ToL1MembershipWitness` resolves the wrong tx and
throws `The L2ToL1Message you are trying to prove inclusion of does not
exist`.

This is hardening, not an incident fix. No honest code path produces two
stored blocks sharing a tx: duplicate nullifiers are rejected at block
building (double-spend validation against the build fork), at validator
re-execution, and at world-state sync (nullifier leaves are
non-updateable). The incident that prompted the investigation turned out
to be a downstream client bug following a v5 behavior change in
`L2BlockSynchronizer`, unrelated to the store. The store should still
not amplify a duplicated-tx state into silent corruption if it ever
appears — e.g. via a future upstream bug, or byzantine tx effects that
repeat a txHash with distinct nullifiers (invisible to every
nullifier-based check).

## Fix

- `removeBlocksAfter` cleans up from the raw storage row, so blocks
whose bodies can no longer be fully loaded still release their row, tx
effects (enumerated from `#blockTxs`), and hash/archive indices.
- `deleteBlock` only deletes a tx-effect entry still owned by the block
being removed (compares the entry's leading block hash).
- Both anomalies now log a warning (an entry owned by another block, or
a missing entry). These states should be unreachable, so a warning in
production logs is direct evidence of an upstream bug worth
investigating — previously the ownership skip was silent, hiding exactly
that signal.

## Cost

The added reads are confined to prune/reorg paths: one `#blockTxs` read
plus one tx-effect point read per tx of each *removed* block (roughly
doubling point reads on block removal). Block ingestion and query paths
are untouched.

## Verification

- Regression test from #24732: two proposed blocks sharing a tx effect;
asserts removal releases both blocks fully with no residue (fails on the
pre-fix implementation).
- Full `block_store.test.ts` suite passes (160 tests).

Recreates #24732 with the original commit cherry-picked and authorship
preserved (thanks @benesjan). On top of it, this PR updates the
description and code comments to match what the follow-up investigation
established about how a duplicated-tx state can and cannot arise, and
adds the anomaly warnings. Supersedes #24732.

Related to A-1426
@AztecBot

Copy link
Copy Markdown
Collaborator

✅ Successfully ported to port-to-next-staging #24964.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

port-to-next Forward-port this merged PR into next

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants