Skip to content

fix(ci): raise AVM check-circuit per-tx timeout to 120s#24952

Draft
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-check-circuit-timeout-r30045391060
Draft

fix(ci): raise AVM check-circuit per-tx timeout to 120s#24952
AztecBot wants to merge 1 commit into
nextfrom
cb/avm-check-circuit-timeout-r30045391060

Conversation

@AztecBot

Copy link
Copy Markdown
Collaborator

What

Raise the per-tx timeout for the AVM check-circuit commands in yarn-project/end-to-end/bootstrap.sh from 30s to 120s, and refresh the stale warning comment.

Root cause (identical to every prior occurrence)

The AVM Circuit Inputs Collection and Check workflow failed on the next merge queue (run 30045391060, commit 6295a38). The avm-check-circuit job exited 124 — a wall-clock timeout, not a correctness failure. From the CI dashboard log, every dumped tx checked in 2–3s except one:

FAILED: bb-avm avm_check_circuit ... single-node/block-building/multiple_blobs/...tx-0x04948c4f....bin (31s) (code: 124)
Running check (with skippable) circuit over 700560 rows.
timeout: sending signal TERM to command 'bash'

avm_check_circuit_cmds applies a uniform TIMEOUT=30s (default CPUS=2/MEM=8g) to every tx. The multiple_blobs BatchCall tx emits a full-size public log to span >1 blob, producing by far the largest AVM circuit in the suite (~700k rows). Simulation + trace-gen (~6s) + checking that circuit (~24s) on 2 CPUs lands right at the 30s ceiling with zero headroom, so it intermittently trips under merge-queue load. No OOM (dmesg clean), no assertion. The triggering commit (#24934) only forward-ports cli/bot/node-keystore and deletes THREAT_MODEL.md — nothing AVM-related, so this is not a regression. The existing in-code WARNING comment anticipated exactly this.

Fix

Raise the ceiling to 120s (~4× the largest observed check). Small txs are unaffected — the timeout is a maximum and they still finish in 2–3s. check_circuit is bounded, deterministic O(rows) work, so a larger ceiling cannot let a hang run forever. CPUS/MEM left at defaults (peak mem was ~2.1 GiB; bumping CPUs would oversubscribe the 64-core runner against the 16-way parallelize); the refreshed comment points maintainers at those knobs if a future tx needs them.

⚠️ This is a chronic auto-dispatch loop — the blocker is the merge, not the fix

This exact failure has been dispatched to claudebox dozens of times; ~30 cb/avm-check-circuit-timeout* branches exist. Prior fix PRs were never rejected on merit — they were auto-closed as stale claudebox drafts after 5 days unmerged, so next still carries TIMEOUT=30s and the job keeps going red and re-dispatching:

  • #23749 — 30s→120s — closed unmerged 2026-06-10
  • #23803 — 30s→120s — closed unmerged 2026-06-08
  • #24257 — 30s→180s — closed unmerged 2026-06-29
  • #23742, #23670 — equivalent — auto-closed stale

This PR needs to be merged (by a maintainer, or landed via claudebox merge) to break the cycle. Pinging the AVM / #team-bonobos owners.

Testing

Config-only change to a CI timeout; no code paths change. The fix is exercised by this PR's own avm-circuit-inputs run (the ci-draft label runs CI on the draft) — the previously-timing-out multiple_blobs check now has headroom to complete.


Created by claudebox · group: slackbot

@AztecBot AztecBot added ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR. labels Jul 23, 2026
@AztecBot

Copy link
Copy Markdown
Collaborator Author

Superseded by #24959, which makes the same avm_check_circuit budget fix, is rebased on current next (so its CI is meaningful), and carries the evidence from the latest failure (run 30266937162).

Five open PRs plus ~50 stale branches all propose this same one-line change and none has ever landed. Consolidating on #24959 — safe to close this one.


Created by claudebox · group: slackbot

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-draft Run CI on draft PRs. ci-no-fail-fast Sets NO_FAIL_FAST in the CI so the run is not aborted on the first failure claudebox Owned by claudebox. it can push to this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant