fix(finalizer): bound Arbitrum withdrawal search at the confirmed send root - #3703
fix(finalizer): bound Arbitrum withdrawal search at the confirmed send root#3703droplet-rl wants to merge 3 commits into
Conversation
…d root arbStackFinalizer bounded its event search at `now - challengePeriodSeconds`, an estimate of when withdrawals ought to have become confirmed. The Outbox only settles a leaf beneath a confirmed send root, so the estimate is a proxy for the real gate rather than the gate itself -- and it is wrong in both directions. Measured on Arbitrum One at 2026-08-11 11:16Z: the estimate put the bound at L2 block 490,993,567 while the Outbox's latest SendRootUpdated covered 491,188,161, a gap of ~195k blocks (~14h). Both bounds advance at chain rate, so the lag is systematic: every Arbitrum withdrawal was discovered ~14h after it became executable. Four SpokePool->HubPool withdrawals (28.29 WBTC) were sitting confirmed-but-undiscovered when this was found, with the finalizer logging "Found 0 Arbitrum One messages ... for finalization" on every run. Read the frontier from the Outbox instead. SendRootUpdated carries the L2 block hash each confirmed root covers, which is exactly what executeTransaction() enforces. The estimate is kept as a fallback for when no confirmation is visible in the lookback (e.g. a halted Orbit chain), and the log line now records which bound was used so a silent regression is visible. Also note the failure mode in the other direction: were assertions ever to confirm slower than the configured challenge period, the estimate would run ahead of the frontier and the finalizer would repeatedly attempt leaves that cannot yet settle. Reading the Outbox is correct either way. Adds SendRootUpdated to the Arbitrum Outbox ABI, which previously carried no events. Applies to all Orbit chains served by arbStack.ts. Co-Authored-By: Claude <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5828815ade
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| const sendRootEvents = await paginatedEventQuery(outbox, outbox.filters.SendRootUpdated(), { | ||
| from: Math.max(LATEST_MAINNET_BLOCK - lookbackBlocks, 0), | ||
| to: LATEST_MAINNET_BLOCK, | ||
| }); |
There was a problem hiding this comment.
Paginate the send-root event lookup
When finalizing Arbitrum One, this lookback spans roughly 50,000 L1 blocks, but omitting maxLookBack makes paginatedEventQuery issue one eth_getLogs request for the entire range. The repository deliberately defaults RPC event ranges to 10,000 blocks in src/common/Constants.ts, so providers enforcing that limit will reject this query; after retries, the catch path silently restores the challenge-period estimate and the withdrawal-discovery delay this commit is intended to fix. Pass the configured mainnet maximum so this lookup is split into supported ranges.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Good catch, fixed in aa6b453.
Confirmed the mechanism: getPaginatedBlockRanges short-circuits to [[from, to]] when maxLookBack is undefined, so the whole range went out as one eth_getLogs. For Arbitrum One the lookback is 604800 / ~12s ≈ 50k L1 blocks, 5x the 10k default in CHAIN_MAX_BLOCK_LOOKBACK.
The consequence is worse than a slow query, which is why this is worth fixing rather than leaving: the rejection would be swallowed by the catch and the function would return undefined, so the caller falls back to the challenge-period estimate — silently reinstating the exact ~14h discovery lag this PR removes, with the bound: "challengePeriodEstimate" log line as the only signal.
Now passes maxLookBack: CHAIN_MAX_BLOCK_LOOKBACK[CHAIN_IDs.MAINNET], matching opStack.ts:875 and linea/l2ToL1.ts:197. Ordering is unaffected — getPaginatedBlockRanges returns ranges ascending and paginatedEventQuery flattens them in order, so .at(-1) is still the newest root.
paginatedEventQuery collapses to a single eth_getLogs when maxLookBack is undefined (getPaginatedBlockRanges returns the whole range unsplit). The send-root lookback is one challenge period of L1 blocks -- ~50k for Arbitrum One's 7 days -- which is 5x the 10k default this repo configures in CHAIN_MAX_BLOCK_LOOKBACK, so providers enforcing that cap would reject the query. The catch path would then swallow the failure and restore the challenge-period estimate, silently reinstating the discovery lag this PR exists to remove. Pass the configured mainnet maximum, matching opStack.ts and linea/l2ToL1.ts. Range order is unaffected: getPaginatedBlockRanges returns ranges ascending and paginatedEventQuery flattens them in order, so .at(-1) is still the newest root. Co-Authored-By: Claude <noreply@anthropic.com>
|
@codex reviewed one P2 — valid, fixed in aa6b453. Paginate the send-root event lookup. The failure mode is what makes it worth fixing rather than deferring: a provider rejection would be caught by the function's own Now passes No doc updates proposed — this is a bounded fix inside an existing finalizer utility; no interface, config, or runtime-flow change that |
Summary
Testing
|
| ): Promise<number | undefined> { | ||
| const at = `Finalizer#${getNetworkName(chainId)}Finalizer`; | ||
| try { | ||
| const { address, abi } = getContractEntry(CHAIN_IDs.MAINNET, `orbitOutbox_${chainId}`); |
There was a problem hiding this comment.
This is fragile because it assumes mainnet. Can we derive the L1 chainId instead?
|
|
||
| // One challenge period of L1 blocks contains at least one confirmation on a live chain. That is ~50k blocks | ||
| // for a 7-day period, so the range has to be paginated to stay under provider eth_getLogs limits. | ||
| const lookbackBlocks = Math.ceil(getArbitrumOrbitFinalizationTime(chainId) / MAINNET_BLOCK_TIME); |
There was a problem hiding this comment.
Is MAINNET_BLOCK_TIME able to be derived instead of assumed?
There was a problem hiding this comment.
Good catch — it isn't. Fixed in 49e6158.
MAINNET_BLOCK_TIME comes from arch.evm.averageBlockTime(), which looks derived but returns a hardcoded seed here. The SDK seeds its cache at module load and returns it while the TTL holds:
const now = getCurrentTime(); // module load
const blockTimes = { [CHAIN_IDs.MAINNET]: { average: 12.5, timestamp: now, ... } };
const cacheTTL = 60 * 15;
// averageBlockTime():
if (isDefined(cache) && now < cache.timestamp + cacheTTL) return { average: cache.average, ... };A finalizer run that starts and finishes inside 15 minutes of process start never touches the RPC — it gets 12.5s.
It's also the wrong constant, in the unsafe direction. Mainnet runs ~12s, so 604800 / 12.5 = ~48.4k blocks ≈ 6.7 days, not the 7 the "one challenge period contains at least one confirmation" invariant depends on. The shortfall is silent: a chain confirming near the period boundary falls out of the window, the function returns undefined, and the caller drops back to the challenge-period estimate this PR exists to replace.
Now resolved from real block timestamps:
const fromBlock = await getBlockForTimestamp(
logger, CHAIN_IDs.MAINNET, getCurrentTime() - getArbitrumOrbitFinalizationTime(chainId), undefined, redis
);getBlockForTimestamp binary-searches actual timestamps and is redis-cached across runs, so the window is a true challenge period regardless of block-time drift — and it's the same helper the fallback path already uses. Costs one extra lookup, cached.
Left MAINNET_BLOCK_TIME alone for its other two callers (confirmPeriodBlocks at L69 and the finalization-ETA log at L433). Both are estimates where a seeded 12.5s is defensible; happy to revisit separately if you'd rather it were measured everywhere.
yarn build + eslint + prettier clean.
|
No action needed — this change is already on the PR, and the follow-up PR should not be opened. I made the same fix in The landed diff is the same fix Codex describes: -import { getContractEntry } from "../../common";
+import { CHAIN_MAX_BLOCK_LOOKBACK, getContractEntry } from "../../common";
+ maxLookBack: CHAIN_MAX_BLOCK_LOOKBACK[CHAIN_IDs.MAINNET],plus a comment noting the ~50k-block range is why pagination is required. Ordering is unchanged, as Codex also notes. One correction for the record: the summary's permalinks point at blob Two housekeeping notes on working-tree state, since Codex flagged its own: mine had unrelated deletions of Not tagging @codex, since there's no new diff to re-review. |
OutcomeNo code changes, commit, or follow-up pull request were created. The requested pagination fix is already present at the current PR head, Checks
|
The send-root search window was sized as challengePeriodSeconds / MAINNET_BLOCK_TIME. That reads as derived, but is not: averageBlockTime() seeds its cache at module load with a hardcoded 12.5s for mainnet under a 15-minute TTL, and returns the seed without touching the RPC while the TTL holds. A finalizer run that starts and finishes inside that window never measures the chain, so the divisor is a constant in practice. It is also the wrong constant. Mainnet runs ~12s, so 604800/12.5 yields ~48.4k blocks, which spans ~6.7 days rather than the 7 the "one challenge period contains at least one confirmation" invariant depends on. The shortfall is silent: a chain confirming near the period boundary drops out of the window, getLatestConfirmedL2Block() returns undefined, and the caller falls back to the challenge-period estimate this PR exists to replace. Resolve the start block with getBlockForTimestamp() against mainnet instead. It binary-searches actual block timestamps and is redis-cached across runs, so the window is a real challenge period regardless of block time drift. This is the same helper the fallback path already uses. MAINNET_BLOCK_TIME is left in place for its other two callers. Co-Authored-By: Claude <noreply@anthropic.com>
Motivation
arbStackFinalizerbounded its event search atnow - challengePeriodSeconds:That estimates when withdrawals ought to have become confirmed. The Outbox only settles a leaf beneath a confirmed send root, so the estimate is a proxy for the real gate rather than the gate itself.
Measured on Arbitrum One, 2026-08-11 11:16Z:
now - challengePeriodSeconds(current)SendRootUpdated(actual)Both bounds advance at chain rate, so this is systematic, not a one-off: every Arbitrum withdrawal was discovered ~14h after it became executable.
This was found with four SpokePool→HubPool withdrawals (28.29 WBTC) sitting confirmed-but-undiscovered, while the finalizer logged
Found 0 Arbitrum One messages (0 withdrawals | 0 deposits | 0 misc txns) for finalizationon every 30-minute run andstatusesGrouped: {EXECUTED: 6}. Nothing was broken — the items were simply past the window's edge:Change
Read the frontier from the Outbox.
SendRootUpdated(bytes32 outputRoot, bytes32 l2BlockHash)fires on every confirmed assertion and carries the L2 block hash the root covers — exactly whatexecuteTransaction()enforces.getLatestConfirmedL2Block()resolves that hash to a block number.TokensBridged event filterlog line now recordsbound: "outboxSendRoot" | "challengePeriodEstimate", so a silent regression to the estimate is visible in prod.SendRootUpdatedtoArbitrumOutbox.json, which previously carried no events.Why not just shrink the constant
Shaving the offset narrows the gap but re-guesses the same unknown, and it errs the wrong way. If assertions ever confirm slower than the configured period, a shorter offset pushes the bound past the frontier and the finalizer repeatedly attempts leaves that cannot settle. Reading the Outbox is correct in both directions.
Scope and risk
arbStack.ts, not just Arbitrum One.eth_getLogs(bounded to one challenge period of L1 blocks) plus oneeth_getBlockByHashper finalizer run.Testing
yarn lintandyarn typecheckclean.I did not add unit coverage — there is no existing
arbStacktest to extend, and exercising this needs a mocked Outbox plus L1/L2 providers. Flagging that rather than implying coverage. The mechanism was verified directly against mainnet: the Outbox's latestSendRootUpdatedat L1 block 25,731,242 resolves to L2 block 491,188,161, which is the number the helper returns.Per
AGENTS.md: noREADME.md/AGENTS.mdupdates proposed —src/finalizer/has no module doc, and the change is an internal correctness fix with no config, interface, or runtime-flow change.🤖 Generated with Claude Code