Skip to content

fix(finalizer): bound Arbitrum withdrawal search at the confirmed send root - #3703

Open
droplet-rl wants to merge 3 commits into
masterfrom
droplet/arb-finalizer-confirmed-frontier
Open

fix(finalizer): bound Arbitrum withdrawal search at the confirmed send root#3703
droplet-rl wants to merge 3 commits into
masterfrom
droplet/arb-finalizer-confirmed-frontier

Conversation

@droplet-rl

Copy link
Copy Markdown
Contributor

Motivation

arbStackFinalizer bounded its event search at now - challengePeriodSeconds:

// Arbitrum orbit takes 7 days to finalize withdrawals, so don't look up events younger than that.
const latestBlockToFinalize = await getBlockForTimestamp(
  logger, chainId, getCurrentTime() - getArbitrumOrbitFinalizationTime(chainId), undefined, redis
);

That estimates when withdrawals ought to have become confirmed. The Outbox only settles a leaf beneath a confirmed send root, so the estimate is a proxy for the real gate rather than the gate itself.

Measured on Arbitrum One, 2026-08-11 11:16Z:

bound L2 block
now - challengePeriodSeconds (current) 490,993,567
Outbox latest SendRootUpdated (actual) 491,188,161
gap 194,594 blocks ≈ 14.4h

Both bounds advance at chain rate, so this is systematic, not a one-off: every Arbitrum withdrawal was discovered ~14h after it became executable.

This was found with four SpokePool→HubPool withdrawals (28.29 WBTC) sitting confirmed-but-undiscovered, while the finalizer logged Found 0 Arbitrum One messages (0 withdrawals | 0 deposits | 0 misc txns) for finalization on every 30-minute run and statusesGrouped: {EXECUTED: 6}. Nothing was broken — the items were simply past the window's edge:

outbox position L2 block blocks past bound
164347 491,004,395 ~11k
164350 491,071,735 ~78k
164355 491,125,869 ~132k
164357 491,200,970 ~207k

Change

Read the frontier from the Outbox. SendRootUpdated(bytes32 outputRoot, bytes32 l2BlockHash) fires on every confirmed assertion and carries the L2 block hash the root covers — exactly what executeTransaction() enforces.

  • getLatestConfirmedL2Block() resolves that hash to a block number.
  • The wall-clock estimate is kept as a fallback for when no confirmation is visible in the lookback (e.g. a halted Orbit chain — Aleph Zero's rollup has not confirmed since 2025-09-16).
  • The TokensBridged event filter log line now records bound: "outboxSendRoot" | "challengePeriodEstimate", so a silent regression to the estimate is visible in prod.
  • Adds SendRootUpdated to ArbitrumOutbox.json, which previously carried no events.

Why not just shrink the constant

Shaving the offset narrows the gap but re-guesses the same unknown, and it errs the wrong way. If assertions ever confirm slower than the configured period, a shorter offset pushes the bound past the frontier and the finalizer repeatedly attempts leaves that cannot settle. Reading the Outbox is correct in both directions.

Scope and risk

  • Applies to every Orbit chain served by arbStack.ts, not just Arbitrum One.
  • Strictly widens the search window, so it cannot cause a withdrawal to be missed relative to today.
  • The lookup is wrapped in try/catch and falls back to current behavior on any failure, so it cannot break finalization.
  • One extra eth_getLogs (bounded to one challenge period of L1 blocks) plus one eth_getBlockByHash per finalizer run.

Testing

yarn lint and yarn typecheck clean.

I did not add unit coverage — there is no existing arbStack test to extend, and exercising this needs a mocked Outbox plus L1/L2 providers. Flagging that rather than implying coverage. The mechanism was verified directly against mainnet: the Outbox's latest SendRootUpdated at L1 block 25,731,242 resolves to L2 block 491,188,161, which is the number the helper returns.

Per AGENTS.md: no README.md / AGENTS.md updates proposed — src/finalizer/ has no module doc, and the change is an internal correctness fix with no config, interface, or runtime-flow change.

🤖 Generated with Claude Code

…d root

arbStackFinalizer bounded its event search at `now - challengePeriodSeconds`,
an estimate of when withdrawals ought to have become confirmed. The Outbox only
settles a leaf beneath a confirmed send root, so the estimate is a proxy for the
real gate rather than the gate itself -- and it is wrong in both directions.

Measured on Arbitrum One at 2026-08-11 11:16Z: the estimate put the bound at L2
block 490,993,567 while the Outbox's latest SendRootUpdated covered 491,188,161,
a gap of ~195k blocks (~14h). Both bounds advance at chain rate, so the lag is
systematic: every Arbitrum withdrawal was discovered ~14h after it became
executable. Four SpokePool->HubPool withdrawals (28.29 WBTC) were sitting
confirmed-but-undiscovered when this was found, with the finalizer logging
"Found 0 Arbitrum One messages ... for finalization" on every run.

Read the frontier from the Outbox instead. SendRootUpdated carries the L2 block
hash each confirmed root covers, which is exactly what executeTransaction()
enforces. The estimate is kept as a fallback for when no confirmation is visible
in the lookback (e.g. a halted Orbit chain), and the log line now records which
bound was used so a silent regression is visible.

Also note the failure mode in the other direction: were assertions ever to
confirm slower than the configured challenge period, the estimate would run
ahead of the frontier and the finalizer would repeatedly attempt leaves that
cannot yet settle. Reading the Outbox is correct either way.

Adds SendRootUpdated to the Arbitrum Outbox ABI, which previously carried no
events. Applies to all Orbit chains served by arbStack.ts.

Co-Authored-By: Claude <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5828815ade

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +238 to +241
const sendRootEvents = await paginatedEventQuery(outbox, outbox.filters.SendRootUpdated(), {
from: Math.max(LATEST_MAINNET_BLOCK - lookbackBlocks, 0),
to: LATEST_MAINNET_BLOCK,
});

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Paginate the send-root event lookup

When finalizing Arbitrum One, this lookback spans roughly 50,000 L1 blocks, but omitting maxLookBack makes paginatedEventQuery issue one eth_getLogs request for the entire range. The repository deliberately defaults RPC event ranges to 10,000 blocks in src/common/Constants.ts, so providers enforcing that limit will reject this query; after retries, the catch path silently restores the challenge-period estimate and the withdrawal-discovery delay this commit is intended to fix. Pass the configured mainnet maximum so this lookup is split into supported ranges.

Useful? React with 👍 / 👎.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, fixed in aa6b453.

Confirmed the mechanism: getPaginatedBlockRanges short-circuits to [[from, to]] when maxLookBack is undefined, so the whole range went out as one eth_getLogs. For Arbitrum One the lookback is 604800 / ~12s ≈ 50k L1 blocks, 5x the 10k default in CHAIN_MAX_BLOCK_LOOKBACK.

The consequence is worse than a slow query, which is why this is worth fixing rather than leaving: the rejection would be swallowed by the catch and the function would return undefined, so the caller falls back to the challenge-period estimate — silently reinstating the exact ~14h discovery lag this PR removes, with the bound: "challengePeriodEstimate" log line as the only signal.

Now passes maxLookBack: CHAIN_MAX_BLOCK_LOOKBACK[CHAIN_IDs.MAINNET], matching opStack.ts:875 and linea/l2ToL1.ts:197. Ordering is unaffected — getPaginatedBlockRanges returns ranges ascending and paginatedEventQuery flattens them in order, so .at(-1) is still the newest root.

paginatedEventQuery collapses to a single eth_getLogs when maxLookBack is
undefined (getPaginatedBlockRanges returns the whole range unsplit). The
send-root lookback is one challenge period of L1 blocks -- ~50k for Arbitrum
One's 7 days -- which is 5x the 10k default this repo configures in
CHAIN_MAX_BLOCK_LOOKBACK, so providers enforcing that cap would reject the
query. The catch path would then swallow the failure and restore the
challenge-period estimate, silently reinstating the discovery lag this PR
exists to remove.

Pass the configured mainnet maximum, matching opStack.ts and linea/l2ToL1.ts.
Range order is unaffected: getPaginatedBlockRanges returns ranges ascending and
paginatedEventQuery flattens them in order, so .at(-1) is still the newest root.

Co-Authored-By: Claude <noreply@anthropic.com>
@droplet-rl

Copy link
Copy Markdown
Contributor Author

@codex reviewed one P2 — valid, fixed in aa6b453.

Paginate the send-root event lookup. paginatedEventQuery was called without maxLookBack, and getPaginatedBlockRanges short-circuits to a single unsplit range in that case. Arbitrum One's lookback is one challenge period of L1 blocks (604800s / ~12s ≈ 50k), 5x the 10k default this repo configures in CHAIN_MAX_BLOCK_LOOKBACK.

The failure mode is what makes it worth fixing rather than deferring: a provider rejection would be caught by the function's own catch, which returns undefined and hands the caller back to the challenge-period estimate — silently reinstating the ~14h discovery lag this PR exists to remove. The bound: "challengePeriodEstimate" log line added here would have been the only trace.

Now passes maxLookBack: CHAIN_MAX_BLOCK_LOOKBACK[CHAIN_IDs.MAINNET], consistent with opStack.ts:875 and linea/l2ToL1.ts:197. Event ordering is unaffected: getPaginatedBlockRanges emits ranges ascending and paginatedEventQuery flattens them in order, so .at(-1) is still the newest root. yarn build and eslint/prettier clean.

No doc updates proposed — this is a bounded fix inside an existing finalizer utility; no interface, config, or runtime-flow change that src/finalizer docs describe.

@droplet-rl

Copy link
Copy Markdown
Contributor Author

🔎 View trace

@chatgpt-codex-connector

Copy link
Copy Markdown

Summary

  • Imported the repository’s configured chain block-lookback limits and applied Ethereum mainnet’s limit to the Arbitrum SendRootUpdated event query. This ensures the challenge-period search is divided into provider-safe ranges instead of issuing one oversized eth_getLogs request. src/finalizer/utils/arbStack.tsL33-L40 src/finalizer/utils/arbStack.tsL236-L242
  • Preserved ascending event ordering and the existing .at(-1) selection of the newest confirmed send root. src/finalizer/utils/arbStack.tsL244-L245
  • Committed the change as 7123d08 (fix(finalizer): paginate Arbitrum send-root lookup).
  • Prepared the follow-up PR titled “fix(finalizer): paginate Arbitrum send-root lookup.”
  • No documentation updates were needed because this is an internal query-bounding correction with no interface, configuration, or documented runtime-flow change.
  • Left the pre-existing, unrelated yarn.lock working-tree modification untouched.

Testing

  • yarn prettier --check src/finalizer/utils/arbStack.ts
  • yarn build
  • yarn lint
  • git diff --check -- src/finalizer/utils/arbStack.ts

View task →

): Promise<number | undefined> {
const at = `Finalizer#${getNetworkName(chainId)}Finalizer`;
try {
const { address, abi } = getContractEntry(CHAIN_IDs.MAINNET, `orbitOutbox_${chainId}`);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is fragile because it assumes mainnet. Can we derive the L1 chainId instead?

Comment thread src/finalizer/utils/arbStack.ts Outdated

// One challenge period of L1 blocks contains at least one confirmation on a live chain. That is ~50k blocks
// for a 7-day period, so the range has to be paginated to stay under provider eth_getLogs limits.
const lookbackBlocks = Math.ceil(getArbitrumOrbitFinalizationTime(chainId) / MAINNET_BLOCK_TIME);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is MAINNET_BLOCK_TIME able to be derived instead of assumed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — it isn't. Fixed in 49e6158.

MAINNET_BLOCK_TIME comes from arch.evm.averageBlockTime(), which looks derived but returns a hardcoded seed here. The SDK seeds its cache at module load and returns it while the TTL holds:

const now = getCurrentTime();                                        // module load
const blockTimes = { [CHAIN_IDs.MAINNET]: { average: 12.5, timestamp: now, ... } };
const cacheTTL = 60 * 15;
// averageBlockTime():
if (isDefined(cache) && now < cache.timestamp + cacheTTL) return { average: cache.average, ... };

A finalizer run that starts and finishes inside 15 minutes of process start never touches the RPC — it gets 12.5s.

It's also the wrong constant, in the unsafe direction. Mainnet runs ~12s, so 604800 / 12.5 = ~48.4k blocks ≈ 6.7 days, not the 7 the "one challenge period contains at least one confirmation" invariant depends on. The shortfall is silent: a chain confirming near the period boundary falls out of the window, the function returns undefined, and the caller drops back to the challenge-period estimate this PR exists to replace.

Now resolved from real block timestamps:

const fromBlock = await getBlockForTimestamp(
  logger, CHAIN_IDs.MAINNET, getCurrentTime() - getArbitrumOrbitFinalizationTime(chainId), undefined, redis
);

getBlockForTimestamp binary-searches actual timestamps and is redis-cached across runs, so the window is a true challenge period regardless of block-time drift — and it's the same helper the fallback path already uses. Costs one extra lookup, cached.

Left MAINNET_BLOCK_TIME alone for its other two callers (confirmPeriodBlocks at L69 and the finalization-ETA log at L433). Both are estimates where a seeded 12.5s is defensible; happy to revisit separately if you'd rather it were measured everywhere.

yarn build + eslint + prettier clean.

@droplet-rl

Copy link
Copy Markdown
Contributor Author

No action needed — this change is already on the PR, and the follow-up PR should not be opened.

I made the same fix in aa6b453, which is the current PR head (gh pr view 3703 → 2 commits: 5828815, aa6b453). Codex's 7123d08 does not exist on the remote — GET /repos/across-protocol/relayer/commits/7123d08 returns 422 — so it's a sandbox-local commit that duplicates work already landed. No PR with that title exists yet either.

The landed diff is the same fix Codex describes:

-import { getContractEntry } from "../../common";
+import { CHAIN_MAX_BLOCK_LOOKBACK, getContractEntry } from "../../common";
+      maxLookBack: CHAIN_MAX_BLOCK_LOOKBACK[CHAIN_IDs.MAINNET],

plus a comment noting the ~50k-block range is why pagination is required. Ordering is unchanged, as Codex also notes.

One correction for the record: the summary's permalinks point at blob 5828815, the pre-fix commit. Those line ranges show the unpaginated query, not the fix — read aa6b453 instead.

Two housekeeping notes on working-tree state, since Codex flagged its own: mine had unrelated deletions of .env.example and src/monitor/.env.sample (artifacts of yarn install in this sandbox). They were never staged, so they didn't reach aa6b453, and I've since restored them. Tree is clean at the PR head.

Not tagging @codex, since there's no new diff to re-review.

@droplet-rl

Copy link
Copy Markdown
Contributor Author

🔎 View trace

@chatgpt-codex-connector

Copy link
Copy Markdown

Outcome

No code changes, commit, or follow-up pull request were created. The requested pagination fix is already present at the current PR head, aa6b453.

Checks

  • git rev-parse HEAD — confirmed aa6b453ba6b611512bea4e5ae9b381e4fbb33b26.
  • git log -2 --oneline — confirmed the pagination fix is the latest commit, following 5828815.
  • ⚠️ git status --short — showed an existing unstaged yarn.lock modification; it was left untouched because it is unrelated to the trigger.

View task →

The send-root search window was sized as challengePeriodSeconds /
MAINNET_BLOCK_TIME. That reads as derived, but is not: averageBlockTime() seeds
its cache at module load with a hardcoded 12.5s for mainnet under a 15-minute
TTL, and returns the seed without touching the RPC while the TTL holds. A
finalizer run that starts and finishes inside that window never measures the
chain, so the divisor is a constant in practice.

It is also the wrong constant. Mainnet runs ~12s, so 604800/12.5 yields ~48.4k
blocks, which spans ~6.7 days rather than the 7 the "one challenge period
contains at least one confirmation" invariant depends on. The shortfall is
silent: a chain confirming near the period boundary drops out of the window,
getLatestConfirmedL2Block() returns undefined, and the caller falls back to the
challenge-period estimate this PR exists to replace.

Resolve the start block with getBlockForTimestamp() against mainnet instead. It
binary-searches actual block timestamps and is redis-cached across runs, so the
window is a real challenge period regardless of block time drift. This is the
same helper the fallback path already uses.

MAINNET_BLOCK_TIME is left in place for its other two callers.

Co-Authored-By: Claude <noreply@anthropic.com>
@droplet-rl

Copy link
Copy Markdown
Contributor Author

🔎 View trace

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants