Repository navigation
epic: faster FPGA builds through .t27/t27c and by removing CI waste #851
Description
Goal
Make each FPGA build faster, in two ways:
- move steps onto
.t27specs compiled byt27c; - remove the waste the current flow already has.
Every claim is measured with negative controls, and every task is an issue below.
Where the time goes today
Railway, idle, 48 CPUs. dayo64 design on xc7a200tfbg484-2, cached chipdb, job wall 39.7 s:
| step | time |
|---|---|
| yosys | 4.6 s |
| chipdb read + sha256 | ~5.5 s |
| nextpnr | 24.2 s |
| fasm2frames | 5.2 s |
| xc7frames2bit | 0.2 s |
Without the chipdb cache the job takes ~210 s, because bbaexport alone is ~160 s.
Laptop, trinet_node0. yosys and nextpnr are from the 2026-10-03 build. The back half was re-timed on 2026-10-04 (UTC) with both tools on the same host, interleaved, at load ~6–10 on 8 CPUs:
| step | time |
|---|---|
| yosys | 12.5 s |
| nextpnr | 56.2 s |
| fasm2frames | 33.0 s (bitwalk --fasm: 0.41 s) |
| xc7frames2bit | 0.204 s (bitwalk --write: 0.239 s) |
An earlier version of this table showed 34.5 s / 2.9 s and 0.23 s / 2.26 s. Those set bitwalk at load ~290 against openXC7 times from a different day (#847).
Weak points
| weak point | issue | |
|---|---|---|
| W4a | CI chipdb cache key carries run_id, so it never hits; ~5 min per FPGA job |
#843 |
| W4b | setup-zig caches fill the 10 GB limit; 0 chipdb entries survive. Measured since: brain-ci ran setup-zig before checkout, so 238 of 238 hits in 25 runs were deleted unused, while its entries took 8.43 of 10.83 GB (#890 reorders). ci.yml's test cache nets −1.7 s per job: over 2 GiB setup-zig saves an empty entry, and a warm entry is reused only on the runner CPU model that saved it (3 of 3 same, 3 of 4 other-model pairs recompiled; runner image ruled out). -Dcpu=x86_64_v3 made entries portable across CPU models in one pair of runs; pinning is the owner's call (#844). The five brain-unit matrix legs shared one key, so 88 of 110 saves were refused; #891 gives each leg its own key (5 of 5 saved). #890 also adds a gate, zig-cache-order, so the old order cannot come back. After #853 merged, its rules kept 7.97 GB at 01:13Z on 2026-10-05, but by 03:54Z the cache was 11.11 GB, 96.7% setup-zig, and the hourly prune had fired 2 times in 9; the chipdb saved on #852's branch was evicted and its next run rebuilt it (2 m 54 s). #895 adds an 8 GB budget that spends only setup-zig's per-run entries off main, and prunes after brain-ci and CI |
#844, #890, #891, #895 |
| W4c | 5 workflows: static chipdb key, floating image (stale-chipdb risk) | #845 |
| W4d | 7 workflows: chipdb built with no cache at all | #846 |
| W8 | bitwalk --write was reported ~10x slower than xc7frames2bit. Re-measured on one host, interleaved: 1.17x (~35 ms per build), byte-identical. The 10x mixed loads and days, and tri devkit flow now times both sides together |
#847 |
| W1/W2 | bitwalk existed only on one macOS laptop, and its digest needed prjxray Python. Measured since: a CI probe builds it for Linux x86_64 from t27 634b3ff and runs it in the openXC7 image, byte-identical to prjxray in every round. The digest imports only the Python standard library. Static glibc beats static musl (musl spends 0.19 s in sys on --write). FASM to frames is 8.6x faster than that image's fasm2frames, which falls back to its slow textX parser. Railway is not timed yet (#848 comment) |
#848 |
| W8b | bitwalk --write grows faster than linearly: far.t27's fdri_index re-walks every earlier group on each line. A table built once from walk_addresses made it linear (148–151 ms against xc7frames2bit's 243–346 ms on CI, byte-identical), and a test checked it against fdri_index on all three parts. FASM to .bit drops from 3167 to 489 ms on that runner. Option 2 makes the change in far.t27 itself: group_frames becomes a COL_FRAMES sum, and far.rs is generated by t27c. It ran at 144–149 ms over three runs, byte-identical, and passes far.t27's tests 10/10 under t27c test-report, while a mutant fails 5. The fix is in t27, so it is proposed there, not made. t27's CI runs test-report on flow.t27 only (t27#6351) |
gHashTag/t27#6348, gHashTag/t27#6351 |
| W12 | tri fpga-build runs whatever branch ~/trinity-fpga is on |
#849 |
| W5 | nextpnr dominates now. Measured since: this build has no --threads flag. On Railway, dayo64's routing went from 23.6 s to ~6 s with router2. On the 8 small CI conformance designs router2 saves ~0.75 s per seed (~0.2% of a job) and lowers Fmax on 60 of 64 seeds, so CI keeps router1 |
#850 |
| W6 | laptop numbers were taken at load ~290, and a comparison across loads produced #847's false 10x. The back half has since been re-timed at load ~6–10 | re-timed in each issue |
| W7 | bitwalk's oracle is byte agreement with prjxray, not with Vivado, on 200T | blocked: needs a Vivado 200T bitstream set |
| W9 | t27c traps gHashTag/t27#5606, gHashTag/t27#5607 and gHashTag/t27#5977 (t27b: structs, strings, enums) are all open (re-read 2026-10-04); they gate any further port | tracked in t27 |
| W10 | bbaexport ~160 s: ~40% Python object churn, ~23% text BBA emission. A t27 port needs gHashTag/t27#5977 | upstream draft only, not sent |
| W11 | JTAG load is host-dominated (9.7 MB .bit at 30 MHz) |
needs the board |
| W13 | On CI, fasm2bit is 1.1% of runner time (31 audit jobs), so bitwalk/.t27 barely moves CI; with the chipdb cached, docker pull (1.73 GB image, median 56 s) and checkout (median 20 s) are ~47% of what is left. Measured since: the pull is layer extraction, not bandwidth (51–53 s from loopback, 50–58 s from Docker Hub); a slim runtime image (119 MB compressed) pulls cold in 3.7 s and builds gf12_clean byte-identically (.bit from the sync word), but lacks bbaexport and is not published; sparse checkout leaves 4,358 of 18,946 files |
#865 |
| W14 | Runner contention: the account's Actions pool is shared by trinity, t27 and this repo (~141 runs queued at 18:31Z 2026-10-04). The nine document gates ran on every push to every branch: 2,160 runs in 7 days, 45% of them duplicates of a PR run or on branches with no PR. Measured since: 102 of 104 push-to-main workflows had no concurrency group, so 8 merges in 3 min queued 142 runs on main (19:06Z). At 20:05Z the pool was full from all three repos (in progress: t27 6, trinity 8, this repo 2 runs; queued 75 / 105 / 23), and a 24 s gate waited 25 min. Measured since: 15 workflows on main ran on push to every branch and on pull_request, twice per PR commit, and 3 skipped a push path on PRs; in 24 h, 351 of 374 push runs on non-main branches came from those 15 |
#867, #869 / #870, #875 / #876 |
| W15 | Gates that fail or pass for a reason other than their claim, so a red is ignored and a green proves nothing. Of the 6 workflows with scheduled runs on main from 2026-09-21 to 2026-10-04, 2 were red on their last run, and no PR showed it (codegen was red once, on 2026-09-29, and green on its 5 runs since; that red was the vsa_simd wall-clock test): receipt-keys 7 of 7 since 2026-09-28, a checker reading header comments as instantiations; brain-ci's weekly report 2 of 2, calling a NotImplemented tri stress --record, and on 2026-09-27 its stress step ended green over 15 compile errors (| tee without pipefail). Also: codegen-corpus could not tell a sample that does not compile from one that never ran; fpga-docker's Build Conscious Image passed on 0 of its 6 runs; brain-ci's PR report ran on always() and read a run cancelled by a newer push as DAMAGE, posting "CRITICAL … cannot be merged" (on #870, run 37228527559). #870's own cancel-in-progress would make that routine. Its 🔒 Merge Gate could never fail: without a status function it ran only when everything upstream had passed, and on 3 of 3 PR runs with a failed report it was skipped. On push, codegen-corpus was red on main 5 times from 09-24 to 10-04 for a reason the commit did not decide: it sampled every 32nd entry of the run's own clean list, and a generator abort counted as clean through a missing or empty file, which moved phi_utils_multi into the sample. 19 baseline entries had never written a usable file; 18 crash on a use-after-free in the Verilog path. Two TRI-NET bitstream workflows have been red on main since 2026-08-02 with no PR showing it: Yosys never read trinet_node_core.v, and a | tee hid the error. With a warm Zig cache, an unchanged test binary keeps its earlier pass and is not run again (Zig 0.16 Step/Run.zig), so a clock- or seed-dependent test stops being re-sampled. |
#871, #874, #877 / #878, #879 / #880, #881 / #882, #883 / #884, #885 / #886, #887, #888 / #889, #890 |
| W16 | The run-syntax gate (tools/check_run_syntax.py) read only the top-level run: text of workflow files. A single-quoted bash -c '...' script inside a step was never parsed (24 on main, all parse today), and composite actions under .github/actions/ were not read at all; #852 adds the first, called by 53 jobs. A change to that action alone also started no workflow (0 PR and 0 push triggers; 1 and 1 after #852's 0c5f415) |
#892 / #893, #854 / #894 |
| W17 | 9 workflow files on main pin regymm/openxc7@sha256:eced1cdd… (25 occurrences, counted with grep -c on 57de4a1f5), whose nextpnr predates nextpnr-xilinx 226609704 and fails DSP48E1 designs with $PACKER_GND_NET. Cross run 37264809032: eced1cdd's Yosys with the current image's nextpnr routes the GF multiplier (496 LUT, 1 DSP, 74.64 MHz); the reverse pairing fails. build-ax7203-bitstream-mul.yml is not pinned: it floats on :latest. Of the 9, only tnf-cost-sweep.yml lists a multiplier, and its default arms are add-only. Repinning or -nodsp is the owner's call |
this issue |
| W18 | tnf-cost-sweep.yml was green on main with no arm ever placed and routed since 2026-08-19: the XDC was written to the runner's /tmp, which the container does not mount, and behind that an IOSTANDARD-only XDC let nextpnr auto-place an output on a GTP pad. Fixed on #897: 5 of 5 arms route (run 37266625622), a run whose router never started now fails the job, and speed wflint checks for a host /tmp path that a docker run reads without a mount (1 hit in 122 workflows on main). The first full run also showed an empty wns field in every arm (fixed on #897), Fmax that is not comparable across the PIPELINE boundary at M = 25, and a refit whose verdicts rest on one residual degree of freedom (#898, owner's call) |
#896 / #897, #898 |
Coverage
tri devkit speed coverage classifies every workflow that builds a chipdb (OK / NEVER_HIT / STALE_RISK / NO_CACHE). Its classifier has 9 fixtures, and 5 of 5 mutants are killed.
| chipdb cache | workflows |
|---|---|
| main 57de4a1 | 1/61 OK (1.6%): 48 NEVER_HIT, 5 STALE_RISK, 7 NO_CACHE |
| #852 30b2919 | 54/61 OK (88.5%): 0 NEVER_HIT, 0 STALE_RISK, 7 NO_CACHE |
| #868 a4421f1 (on #852) | 56/61 OK; the other 5 carry a # chipdb-uncached: <why> comment, so 61/61 decided |
Target: 61/61, or a written reason in each workflow that stays uncached. #868 reaches it.
tools/check_concurrency_groups.py (in #870) checks that every push-to-main workflow has a concurrency group that a dispatch cannot cancel.
| concurrency group | push-to-main workflows |
|---|---|
| main 57de4a1 | 2/104 (the gate exits 1, 102 ungrouped) |
| #870 066bfa2 | 105/105 (the gate exits 0; its own run 37228527581 is green) |
tri devkit speed doubles (16 fixtures) classifies each workflow's on: block: DOUBLE_ON_PR (push on every branch plus pull_request), PR_BLIND (push on main only, and a push path missing from the PR paths), DOUBLE_GLOB (push on globbed branches plus pull_request), PUSH_ALL_ONLY.
| once per PR commit | OK | DOUBLE_ON_PR | PR_BLIND |
|---|---|---|---|
| main 57de4a1 | 99/122 (81.1%) | 15 | 3 |
| #876 9477f0f | 108/122 (88.5%) | 9 (the nine #867 changes) | 0 |
| #876 + #867 (sum of the two, not run on a merged tree) | 117/122 | 0 | 0 |
The other 5 are 4 DOUBLE_GLOB (brain-ci, codegen, fpga-formal, fpga-formal-comb) and build-paper (PUSH_ALL_ONLY). Whether those branch pushes are wanted without a PR is the owner's call (#875).
Not measured
- The CI minutes lost per week in total. The per-step figures come from the 31 audit jobs in ci: once the chipdb is cached, docker pull + checkout are ~47% of an FPGA job; bitwalk can save ~1% #865 (2026-10-04), one commit, hosted runners only.
- yosys and nextpnr on an idle laptop. Only the back half was re-timed (load ~6–10 on 8 CPUs).
- bitwalk on Railway (fpga: build bitwalk for Linux x86_64 from t27 master and use it on Railway/CI #848 part 3), the table arm on macOS arm64, and any design larger than blinky.
Related to #14, #842. Merged on 2026-10-04: #842, #853, #857, #861, #863, #864. Open: #852, #866, #867, #868, #870, #872, #873, #876, #878, #880, #882, #884, #886, #889, #890, #891, #893, #894, #895, #897. #887 (a compiler fix) is the owner's call. Merge #872 first: main has been red on doc-refs, orphan-artefacts and gates-can-fail since #510, and every PR inherits that red until #872 lands. Merge #882 before #870 and before #884, #890 before #891, and #893 before #894. #852 and #897 conflict in tnf-cost-sweep.yml (mechanical); the second one to merge gets a rebase.
Activity
gHashTag commentedon Oct 4, 2026
Who else works on fast open 7-series builds
Re-read live on 2026-10-04 with
gh api..bitlromor/fpga-assembler(fpga-as, C++/Bazel).bitis not byte-identical toxc7frames2bit. Our #56 there (case 05) is open with no review. bitwalk is the only other fast path we know of, and it writes ECC.fasm2framesfasm2framesmay predate it.openXC7/nextpnr1.0.0 (himbaechel)nextpnr-xilinxis archivedcavearr/toolchain-openxc7-releases,FPGAwars/tools-openxc7upstream-2026-10-04.binwith achipdb-id.txtstamp; their CI reuses bins when the stamp matches. This is the same idea as the image-ID key in #852..bitfilmil/bazel_rules_openxc7.bitheader, which carries the wall clock and input path, so builds are hermetic.Rust: we found no Rust FASM →
.bitassembler for 7-series (repository search, 2026-10-04).The gap no one else fills: an ECC-correct fast writer, cross-checked frame by frame against prjxray, from a spec. That is bitwalk.
New tasks from this: #854 (gate coverage), #855 (himbaechel). Full notes with every URL are kept in the speed-loop state.
ci(brain): let the merge gate fail, and fail on a failed detect-build
test(codegen): probe 2 for no-output specs and repeat flips (throwaway,
style(brain): name the bool the detect-build verdict tests
gHashTag commentedon Oct 5, 2026
S61: the repo's own fast gates on the tree with every clean open PR merged.
Every open PR is green against main on its own. That does not show they are green together. I trial-merged the open PRs in the proposed order with
speed mergeorder, which usesgit merge-treeandcommit-tree, so no branch and no ref is written. Then I ran each tree-local gate on the result.d746ad6c6. Built on main 57de4a1 with every head pinned by SHA in the order file, so a branch that moves is reported, not silently picked up.b4ef41a4c. Trial commits now carry a fixed identity and date, so the same order gives the same SHA on any machine. When I first posted this it namedda3967f65, which has the same tree but a wall-clock date, so nobody else could reproduce that SHA.doubles: runs once per PR commit and tests every push pathwflintfindingsactionpaths: openxc7-chipdb callers that fire on a PRThe main column was measured on main's
.github/at 57de4a1, extracted withgit archive, running the same commands.One anomaly, found and fixed. The first merged tree had 176 wflint findings. The extra one was mine: #884 added
if [ "$detect_ok" != true ], which breaks the named-bool form. It is fixed in 7fbeb9b. Behaviour is identical for all 5 detect-build results (old and new snippet underbash -e). The table above is the re-run after the fix.A correction to my own count. Earlier I wrote that DOUBLE_GLOB was 4. That is main's count, not the fully merged tree's. On the merged tree it is 5. The fifth is ax7203-led-onehot.yml, which comes from #852:
.github/actions/openxc7-chipdb/**, by design, because it is the one build that exercises the shared action.feat/**push trigger, so a feat/ branch PR that changes the action runs it twice.That is narrow and I propose no change. The other four (brain-ci, codegen, fpga-formal, fpga-formal-comb) are measured in #875.
Not measured
.github/,tools/and a few test inputs.zig buildof anything on the merged tree. The codegen corpus was measured only on the fix(codegen): say why a sampled spec does not compile, and count a compiler that cannot run apart #873+fix(codegen): sample the corpus compile check from the baseline #886 resolution, not on all 18 merged.tools/andgit ls-files, and the extract has only.github/. I did not re-run them locally for this table.Command, after
git fetch:--diffaddsspeed treediff BASE MERGED. It runs coverage, concurrency, doubles and wflint on both commits'.github, read withgit archiveand no checkout, and prints only the findings the merge adds. On this order it prints one finding: the ax7203-led-onehot DOUBLE_GLOB. On the tree from before the #884 fix it printed both the DOUBLE_GLOB and the wflint finding. It takes about 1 s. It has 7 fixtures and 10/10 mutants killed.To run the gates, detach a worktree at the printed final commit.
audit_workflow_paths.pyreadsgit ls-files, so an extracted copy is not enough.🤖 Generated with Claude Code
ci(chipdb): cache two more workflows, state why five probes stay unca…
fix(codegen): say why a sampled spec does not compile, on top of the …
ci: give the 102 push workflows a dispatch-safe concurrency group
ci(brain): let the merge gate fail, and fail on a failed detect-build
style(brain): name the bool the detect-build verdict tests
test(codegen): determinism probe for vibee_gen (throwaway, #851 W15)
test(codegen): probe 2 for no-output specs and repeat flips (throwaway,
ci: run nine workflows once per PR commit, on every path their push r…
ci(brain): let the merge gate fail, and fail on a failed detect-build