fix(node): warm KZG trusted setup at startup - #24775
Merged
PhilWindle merged 2 commits intoJul 27, 2026
Merged
Conversation
The lazy KZG singleton (getKzg) builds its precomputation tables synchronously on first use, blocking the event loop for ~2s locally and 12-15s under production CPU limits. On an RPC node the first use is the archiver reconstructing blobs or the proposal handler uploading them, right after a checkpoint arrives, so the stalled loop overruns the gossipsub mcache window and attestation forwarding is skipped. Warm it in the node factory alongside the bb.js singleton, before any subsystem runs, keeping the cost off the gossip path.
Move the trusted-setup load timing and logging out of the node factory and into getKzg itself, gated on an optional logger argument, using elapsedSync for the measurement. Keeps the factory call site to a plain getKzg(log) and makes the timing available to any warm-up caller.
PhilWindle
approved these changes
Jul 27, 2026
PhilWindle
enabled auto-merge
July 27, 2026 10:32
AztecBot
pushed a commit
that referenced
this pull request
Jul 27, 2026
## Context An RPC node logged `Gossip validation for checkpoint_attestation took 15533ms, approaching mcache eviction window of 8400ms` shortly after startup. Investigation (see A-1425) showed the validations were not slow — the Node.js event loop was fully blocked for 12–15s, and the attestations were queued behind it. The blocking call is `getKzg()`, which lazily builds the `@crate-crypto/node-eth-kzg` precomputation tables synchronously (~2s locally, 12–15s under the pod's CPU limits) on first use. On a plain RPC node that first use is the archiver reconstructing blobs during sync, or the proposal handler uploading blobs right after a checkpoint arrives — both immediately adjacent to gossip traffic, so the stalled loop overruns the gossipsub mcache window (8.4s) and attestation forwarding is skipped. Sequencers already avoid this via `Sequencer.init()`, so only non-sequencer nodes were affected. ## Approach Warm the KZG singleton in `createAztecNodeService`, right after the existing bb.js WASM singleton init and before any subsystem runs. The cost is unchanged (every full node reconstructs blobs and pays it eventually) — it just moves off the gossip path to a point where blocking the loop is harmless. `getKzg()` is an idempotent per-process singleton, so this is a no-op for sequencers and for tests that pre-warm via `warmBlobKzg` (all e2e setups do), and adds no work for pure unit tests, which never construct a full node. Fixes A-1425
Collaborator
|
✅ Successfully ported to port-to-next-staging #24964. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
An RPC node logged
Gossip validation for checkpoint_attestation took 15533ms, approaching mcache eviction window of 8400msshortly after startup. Investigation (see A-1425) showed the validations were not slow — the Node.js event loop was fully blocked for 12–15s, and the attestations were queued behind it. The blocking call isgetKzg(), which lazily builds the@crate-crypto/node-eth-kzgprecomputation tables synchronously (~2s locally, 12–15s under the pod's CPU limits) on first use.On a plain RPC node that first use is the archiver reconstructing blobs during sync, or the proposal handler uploading blobs right after a checkpoint arrives — both immediately adjacent to gossip traffic, so the stalled loop overruns the gossipsub mcache window (8.4s) and attestation forwarding is skipped. Sequencers already avoid this via
Sequencer.init(), so only non-sequencer nodes were affected.Approach
Warm the KZG singleton in
createAztecNodeService, right after the existing bb.js WASM singleton init and before any subsystem runs. The cost is unchanged (every full node reconstructs blobs and pays it eventually) — it just moves off the gossip path to a point where blocking the loop is harmless.getKzg()is an idempotent per-process singleton, so this is a no-op for sequencers and for tests that pre-warm viawarmBlobKzg(all e2e setups do), and adds no work for pure unit tests, which never construct a full node.Fixes A-1425