fix(ci): raise AVM check-circuit per-tx timeout to 120s#24952
Draft
AztecBot wants to merge 1 commit into
Draft
Conversation
Collaborator
Author
|
Superseded by #24959, which makes the same Five open PRs plus ~50 stale branches all propose this same one-line change and none has ever landed. Consolidating on #24959 — safe to close this one. Created by claudebox · group: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Raise the per-tx timeout for the AVM
check-circuitcommands inyarn-project/end-to-end/bootstrap.shfrom30sto120s, and refresh the stale warning comment.Root cause (identical to every prior occurrence)
The AVM Circuit Inputs Collection and Check workflow failed on the
nextmerge queue (run 30045391060, commit6295a38). Theavm-check-circuitjob exited124— a wall-clock timeout, not a correctness failure. From the CI dashboard log, every dumped tx checked in 2–3s except one:avm_check_circuit_cmdsapplies a uniformTIMEOUT=30s(defaultCPUS=2/MEM=8g) to every tx. Themultiple_blobsBatchCalltx emits a full-size public log to span >1 blob, producing by far the largest AVM circuit in the suite (~700k rows). Simulation + trace-gen (~6s) + checking that circuit (~24s) on 2 CPUs lands right at the 30s ceiling with zero headroom, so it intermittently trips under merge-queue load. No OOM (dmesg clean), no assertion. The triggering commit (#24934) only forward-portscli/bot/node-keystoreand deletesTHREAT_MODEL.md— nothing AVM-related, so this is not a regression. The existing in-codeWARNINGcomment anticipated exactly this.Fix
Raise the ceiling to
120s(~4× the largest observed check). Small txs are unaffected — the timeout is a maximum and they still finish in 2–3s.check_circuitis bounded, deterministic O(rows) work, so a larger ceiling cannot let a hang run forever.CPUS/MEMleft at defaults (peak mem was ~2.1 GiB; bumping CPUs would oversubscribe the 64-core runner against the 16-wayparallelize); the refreshed comment points maintainers at those knobs if a future tx needs them.This exact failure has been dispatched to claudebox dozens of times; ~30
cb/avm-check-circuit-timeout*branches exist. Prior fix PRs were never rejected on merit — they were auto-closed as stale claudebox drafts after 5 days unmerged, sonextstill carriesTIMEOUT=30sand the job keeps going red and re-dispatching:This PR needs to be merged (by a maintainer, or landed via claudebox merge) to break the cycle. Pinging the AVM / #team-bonobos owners.
Testing
Config-only change to a CI timeout; no code paths change. The fix is exercised by this PR's own
avm-circuit-inputsrun (theci-draftlabel runs CI on the draft) — the previously-timing-outmultiple_blobscheck now has headroom to complete.Created by claudebox · group:
slackbot