Repository navigation
ci: once the chipdb is cached, docker pull + checkout are ~47% of an FPGA job; bitwalk can save ~1% #865
Description
Part of #851.
What was measured
Step timings of every job in the 31 workflow_dispatch runs of the #862 audit (commit 38da24d, 2026-10-04 15:14–15:31 UTC, GitHub-hosted ubuntu-latest). Source: gh run view <id> --json jobs, one file per run, then a script that puts every step into one bucket by name. 31 of 31 runs read, 31 jobs.
Positive control on the bucketing: every step name that matched no bucket is printed. There were 25 such steps (Run actions/cache@v4 ×21, the HSLM weight steps ×3, Pick the cell), and none took more than 1 s.
| bucket | jobs | median s | min s | max s | median share of job |
|---|---|---|---|---|---|
| chipdb build (cache miss in every job) | 31 | 204 | 0 | 221 | 54% |
| nextpnr | 31 | 62 | 0 | 192 | 18% |
| docker pull | 31 | 56 | 54 | 88 | 17% |
| checkout | 31 | 20 | 10 | 25 | 5% |
| Yosys | 31 | 5 | 1 | 41 | 1% |
| fasm2frames + xc7frames2bit | 31 | 4 | 0 | 9 | 1% |
| setup, post, upload, verify, other | 31 | 4 together | 1% |
Job total: median 349 s, min 97 s (HSLM, failed at Yosys), max 491 s. Sum over the 31 jobs: 11,144 runner-seconds, of which chipdb 5,757 (52%), nextpnr 2,326 (21%), docker pull 1,943 (18%), checkout 571 (5%), Yosys 209 (2%), fasm2bit 127 (1%).
What this says about #851's own thesis
On CI, the .t27/bitwalk path can only touch the fasm2bit bucket: 127 of 11,144 runner-seconds, 1.1%. Replacing fasm2frames with bitwalk saves at most ~4 s per job there. It still matters where fasm2frames is slow (laptop under load: 34.5 s; Railway idle: 5.2 s of a 39.7 s job, 13%), so #848 stays open, but it is not a CI item.
Take the chipdb build away (#852) and 5,387 runner-seconds remain, a mean of ~174 s per job: nextpnr 43%, docker pull 36%, checkout 11%, Yosys 4%, fasm2bit 2%. Two of the three biggest items are fixed overhead that no design change touches.
The image
regymm/openxc7@sha256:35c91073… is 1,732,282,223 bytes compressed in 11 layers (registry manifest, read 2026-10-04):
| layer | compressed | contents (from the image config history) |
|---|---|---|
| apt | 506 MB | build-essential, clang, bison, flex, cmake, gfortran, tcl-dev, python3-dev, … |
| Yosys | 427 MB | git clone of yosys v0.68 with submodules, plus its build tree |
| prjxray | 313 MB | clone with submodules and build |
| nextpnr-xilinx | 266 MB | clone of 1743d0f with submodules (incl. nextpnr-xilinx-meta) and build |
| pip | 169 MB | prjxray Python requirements |
| base, bison, venv, misc | 51 MB |
A job runs yosys, nextpnr-xilinx, fasm2frames (Python) and xc7frames2bit, against one part's database. Most of those bytes are compilers, source trees and build directories.
Checkout
actions/checkout with fetch-depth: 1 spends ~15 s in git fetch and then writes 18,949 files (repo size 897 MB per the API). A conformance job reads its design directory under fpga/ and specs/fpga/constraints/.
Cache pressure, seen while measuring
The 31 audit runs saved 27 chipdb entries on main keyed with github.run_id (chipdb-xc7a200tfbg484-2-regymm[-openxc7]-<run id>), 71 MB each, ~1.9 GB, none of which a later run can hit (#843). Repository cache usage read at the same time: 10.59 GB in 156 entries, above the 10 GB limit, so entries are being evicted (#844, #853).
Candidate fixes, none measured yet
- chipdb: fix(ci): key the chipdb cache on the openXC7 image so it can hit #852. Measured there: 3 m 19 s → 1 s on a second run.
- nextpnr: router2 (fpga: measure nextpnr router2 / --threads now that place-and-route dominates #850). On Railway, dayo64 with HeAP took a median 23.6 s per run with router1 and 6.0–6.4 s with router2; that bitstream is not board-tested. The hosted-runner A/B on the 8 ci(fpga): audit the 31 workflows that pass --placer sa against today's image #862 designs is run 37219231340.
- image: a runtime-only image built
FROM regymm/openxc7@sha256:35c91073…, copying the four tools, their shared libraries, the Python packages fasm2frames imports, and the 200T database, into a slim base. Before claiming anything: compressed size,docker pulltime on a hosted runner, and a positive control that a Blinky.bitbuilt in it is byte-identical to one built in the full image. - checkout:
sparse-checkoutof the design directory andspecs/fpga/constraints(actions/checkout then fetches blobless). Measure the step on the same runner type, 5 repeats each way. - dead cache entries: the 27 per-run chipdb entries on
mainare the kind ci(cache): prune Actions cache entries nothing can restore #853 prunes; removing them frees ~1.9 GB. Not deleted by hand here.
Not measured
- Any of the five fixes above.
- Whether the 56 s pull is bandwidth or layer extraction. The two would point at different fixes.
- Runs other than these 31; queue time before a job starts (not part of any step).
- Self-hosted runners, which may already hold the image.
Activity
[-]ci: after the chipdb cache, docker pull and checkout are a third of each FPGA job; bitwalk can save 1%[/-][+]ci: once the chipdb is cached, docker pull + checkout are ~47% of an FPGA job; bitwalk can save ~1%[/+]gHashTag commentedon Oct 4, 2026
Items 3 and 4 measured: slim runtime image and sparse checkout (S30)
Branch
exp/slim-image. Valid run: 37224577650 (5338870), hostedubuntu-latest, one job. An earlier run, 37223385356, gave the same byte-identity results but reported slim pulls of 0.0 s. That was a harness defect, not a speedup:docker image pruneleaves layers that BuildKit's cache still references, and the slim image had been built on the same runner. The harness now also runsdocker builder prune -af, and countsAlready existsagainstPull completein every pull's own output. A pull with any layer already present is reported ascold=falsewith an ANOMALY line, never as a time.Image
regymm/openxc7@sha256:35c91073…FROMthe full image's base layer + oneADD)localhost:5000on the runner, 3 repsThat answers the open question in this issue's body: the ~56 s pull is layer extraction, not bandwidth. A pull over loopback takes as long as one from Docker Hub.
The slim image holds 2,608 files. They were chosen three ways:
ldd, transitively, for the 53 ELF files; a Python audit hook recording what fasm2frames imports and opens; and, for the 200T database, every top-level per-tile-type file (so that another design's tile types are covered),mapping/, and the 42 part and fabric files fasm2frames opened. The database kept is 83 MB of 194 MB. Under the audit hook, fasm2frames wrote the same frames as without it.Same output, with controls
Design
gf12_clean_ax7203, seed 1, built in both images:b40ccd99…)3646e275…)ae319ef7…).bitAA995566sync word (offset 176) identical (e2b5ab55…); header differs only in the frames path and the UTC build time that xc7frames2bit writes.bitpayload, so identity is not trivially true.bbaexport, so it cannot build a chipdb. It only works where the chipdb cache (fix(ci): key the chipdb cache on the openXC7 image so it can hit #852) hits.Per-step times are the same in both images: Yosys 5.3–6.5 s, nextpnr 6.4–6.9 s, fasm2frames 4.3 s, xc7frames2bit 0.4 s.
Sparse checkout
specs/fpga/constraints).gitWhat it would buy (estimate, not measured)
The census above put docker pull at a median 56 s of a 349 s job. Where the chipdb cache hits, a slim image would cut that to about 4 s, roughly 48 s per job, or ~1,500 of the 11,144 runner-seconds of the 31-job audit. That arithmetic assumes the slim image is pulled from a registry as fast as from loopback, which is not measured.
Not measured
actions/checkout.gf12_clean_ax7203, and any part other than xc7a200tfbg484-2.bbaexport, so a workflow would need both images, or a chipdb step that falls back to the full one.Refs #851 (W13)
🤖 Generated with Claude Code
ci(exp): slim openxc7 image and sparse checkout against the #865 over…