Repository navigation
Circuit-sweep (program-plonky2) in CI vs manual pre-release — decide + implement #50
Description
Activity
Recommendation: Option D (path-filter), required on PRs to
main, withci:circuit-sweeplabel escape-hatchAdd a single
circuit-sweepjob to.github/workflows/ci.yamlthat runs on[self-hosted, m3-ultra]and is triggered by either:paths:filter matchingprogram-plonky2/**on the PR, or- The PR label
ci:circuit-sweep(manual opt-in for the cases where server-side code changes the witness layout — see below).
Mark it required on PRs targeting
main(the release PR); advisory on PRs targetingdevelop.Why D over B / C / E
Witness-layer risk is real but narrow. The boundary is
program-plonky2/src/inputs.rs::ProgramInputsplus theHashOut<F>↔[u8;32]conversions inshared/server(digest_from_bytes/digest_to_bytes,MMR_PROOF_PATH_LENextension, limb packing for the BIP-340 pubkey). A server-side change that re-orders a field, miscounts MMR siblings, or flips endianness on a digest can break proving without touching circuit code. That's the case Option D alone misses — and it's exactly what theci:circuit-sweeplabel escape-hatch is for. Reviewer applies it when a PR touchesshared/src/lib.rs,server/src/state.rs, or anything that crosses the witness boundary.Runner saturation kills E (B+C) and pure B. The M3 Ultra is single-tenant,
--test-threads=1, ~50 GB RAM/thread (scripts/ci-runner/README.md§ "Disk + RAM headroom"). A scheduled nightly sweep collides with the next morning'sci:fullruns; a label-gated sweep on every release PR pays multi-hour wall-time as a blocker every time. Path-filtering keeps the sweep off the queue for the 90% of PRs that touch onlyserver// docs / CI.Release cadence + forgetfulness data point.
develop→mainhas merged exactly once since this repo's lifetime (PR #16, pre-Plonky2). The manual "run sweep before release-PR" convention inCONTRIBUTING.mdhas therefore never been exercised in practice — the Plonky2 path has shipped to PRD zero times. Codifying it as CI before the first real release is cheap insurance against forgetting on attempt #1.Implementation (single PR)
.github/workflows/ci.yaml: newcircuit-sweepjob.runs-on: [self-hosted, m3-ultra],timeout-minutes: 360,if:guarded onpathsfilter ORcontains(labels.*.name, 'ci:circuit-sweep')ORgithub.base_ref == 'main'. Step body:cd program-plonky2 && cargo test --release --lib -- --test-threads=1. PATH-prepend trick fromserver-testsreused.- Branch protection (separate UI op, not in the PR): add
Circuit Sweep (M3 Ultra)as required onmainonly. CONTRIBUTING.md§ Setup: drop the "run circuit sweep manually before release-PR" instruction; replace with a one-liner pointing at the new job + theci:circuit-sweeplabel for witness-boundary PRs.scripts/ci-runner/README.md§ Operations: add a "Circuit Sweep" subsection — expected wallclock (multi-hour), single-runner queue impact, how to cancel viagh run cancelif a release PR needs to ship around a known-good sweep.program-plonky2/CONTRIBUTING.md§ "CI integration": flip "NOT in CI" → "in CI behind path-filter + label".
Follow-ups not solved here
- Failure semantics on release PR: required check = hard block on merge. We'll need a documented override path for the case where the sweep is red on a non-circuit-related flake (orphan-test-binary OOM per
feedback_cleanup_test_binaries.md, runner disk pressure). Proposal: re-run viagh run rerun, not bypass. If a circuit regression genuinely blocks a hotfix release, revert the offending develop commit rather than override the gate. Track as a separate issue. - Caching strategy: shared
target/perscripts/ci-runner/README.md§ Workspace cache means a green sweep can flip red on stale incremental state. Mitigation: dedicated_workdirectory for the circuit-sweep job, separate fromserver-tests. Out of scope here. - Runtime growth: if future Stage 5d-next-N work bumps
INNER_PAD_BITSfurther, sweep wallclock grows superlinearly. Revisittimeout-minutes: 360when that happens.
Second-best: Option B (label-gated, required on release PR)
Considered and rejected. Pros: same blocking semantics, no
paths:heuristic to maintain. Cons: requires PR authors to remember the label on every circuit-touching PR — that's exactly the forgetfulness failure mode this issue is trying to eliminate, just relocated from "remember to run the sweep" to "remember to apply the label". Path-filter automates the common case (circuit PR) and keeps the label as a deliberate, documented escape-hatch for the rare witness-boundary case where the heuristic is wrong. B becomes the right answer ifpaths:filtering proves too noisy in practice (e.g. ifCargo.lock-only changes start triggering the sweep); revisit then.- added 4 commits that reference this issue
on May 20, 2026
Background
The full cyclic-recursion sweep for
program-plonky2(cargo test -p zkcoins-program-plonky2 --release --lib -- --test-threads=1) is not in CI today. It was deliberately kept out of the issue #40 migration: at production parameters (MAX_IN_COINS = 8) the sweep wallclocks in hours, single-runner serialization on dfx01 means running it on every push would saturate the M3 Ultra and starve normal PR-tests + Coverage Gate jobs.The current convention (documented in
CONTRIBUTING.md § Setupandscripts/ci-runner/README.md) is: run it manually before any release-PR-to-main.This issue tracks the decision on whether / how to formalize that.
Options
A. Stay manual
Pros: zero runner load. Status quo.
Cons: easy to forget. No record of what was verified for each release.
B. Dedicated
ci:circuit-sweepPR labelSame pattern as #48 (
ci:full). Authors opt in; CI runs the sweep as an extra job when the label is set. Required only on the release-PR-to-main, enforced by a checklist item or a separate gate.Pros: explicit opt-in, leaves a CI record, can be required on release PRs only.
Cons: each release PR pays the multi-hour wall-time as a blocker.
C. Scheduled run on develop
schedule:trigger (e.g. nightly or weekly). Sweep runs out-of-band on the latest develop. Result published as a status check / commit status.Pros: catches regressions automatically; doesn't block PRs.
Cons: scheduled runs on develop don't gate PR merges by default — drift between "develop sweep last ran at SHA X" and the PR's HEAD.
D. Path-filter on
program-plonky2/**Sweep runs only when the PR touches circuit code (
program-plonky2/**). Most PRs (server/docs/CI) skip it entirely.Pros: targeted; circuit-code PRs get the sweep automatically without manual opt-in.
Cons: still blocks circuit-code PRs for hours. Server-side changes that interact with circuit behaviour at the witness layer wouldn't trigger it.
E. Combination
E.g. D + C: path-filter on
program-plonky2/**for circuit PRs + nightly scheduled on develop as backstop. Or B + C: opt-in label + nightly schedule.Recommendation needed on
Implementation surface (whichever option lands)
.github/workflows/ci.yaml: add acircuit-sweepjob behind the chosen trigger (label / path-filter / schedule).runs-on: [self-hosted, m3-ultra],--test-threads=1,timeout-minutesbumped to ~360 to be safe.scripts/ci-runner/README.md§ "Operations": document the new job + its expected wallclock.CONTRIBUTING.md § Setup: drop the "manual sweep before release-PR" instruction (or refine it depending on option).Refs