Repository navigation
ci(fpga): audit the 31 workflows that pass --placer sa against today's image #862
Description
ax7203-gf10-sub was green in July and red on 2026-10-04 with no change to its inputs. The cause was the --placer sa bug that openXC7/nextpnr-xilinx#162 documents. On this design HeAP placed 24 of 24 seeds, and SA placed 11 of 40 (#856, #861).
31 other workflows on main pass --placer sa, and 29 of them have not run on main since at least 2026-08-05. The oldest of the last 1000 main runs is from that date. They run only when their own paths change. Their state on today's regymm/openxc7:latest (sha256:35c91073) is unknown, which is exactly the state gf10-sub was in.
Step 1: measure (started 2026-10-04 15:17 UTC)
Each of the 31 has been dispatched once on main. All 31 run with no side effects: an artifact upload only, with no push, release, write permission or secret. The results go in the table below.
| workflow | kind | seed search | last main run before this | today |
|---|---|---|---|---|
| ax7203-gf10-add | conformance | 8 | success 2026-08-05 | |
| ax7203-gf14-add | conformance | 8 | success 2026-08-05 | |
| ax7203-gf{4,6,8,12,14,16,24}-sub | conformance | 8 | none since 2026-08-05 | |
| ax7203-gf{4,6,8,12,16,24}-mul-clean | conformance | 8 | none since 2026-08-05 | |
| ax7203-gf{4,6,8,12,16,24}-clean | conformance | 8-16, or another loop form | none since 2026-08-05 | |
| ax7203-corona-decode | conformance | other loop form | none since 2026-08-05 | |
| ax7203-identity-echo | design check | 16 | none since 2026-08-05 | |
| ax7203-blinky-bitstream, ax7203-led-onehot, fpga-hslm-bitstream | anchor, mentions Fmax | fixed seed 1 | none since 2026-08-05 | |
| ax7203-gf16-conformance, ax7203-dual-clk-heartbeat, ax7203-rx-echo-hb, ax7203-uart-loopback, ax7203-uart-tx-probe | mentions Fmax | 8 | none since 2026-08-05 |
"kind" comes from the workflow name and from whether the file mentions Fmax. That is a first cut, not a reading of each file.
Step 2: act per workflow, not by sweep
- Red conformance workflow: repeat ci(fpga): seed searches hide nextpnr's log when every seed fails #856's round three for it on an experiment branch, with SA and HeAP, every seed. Switch it only if HeAP is clean, as ci(fpga): place gf10-sub with HeAP, not SA #861 does for gf10-sub.
- Workflow that records or quotes Fmax: the placer is part of the recorded configuration. Commits 5f571f9, d6e6978 and 30a230b show the placer/router pair moving Fmax by up to 4.3x and changing rankings. Do not switch. If it is red, record why and decide separately (pin the image digest, re-run, or retire it).
- Green workflow: leave it alone. Note how many seeds it needed, once ci(fpga): print nextpnr's log when a seed search finds no clean seed #857 makes that visible.
Not in scope
- The 6 workflows that already pass
--placer heapand the 23 that pass no--placer(defaultheapin a WITH_HEAP build,arch.cc:1390-1394at 1743d0f4). - The floating
:latesttag, which affects every workflow.
Activity
ci(exp): SA/HeAP A/B for the three red --placer sa conformance workflows
fix(ci): give fpga-hslm Yosys the embedding_weights.mem it reads
gHashTag commentedon Oct 4, 2026
Step 1 measured: 27 green, 4 red
All 31 dispatches on main (
38da24d3a, imageregymm/openxc7:latest=sha256:35c91073) ran 15:12-15:32 UTC. "Seed" is the seed the search stopped on, i.e. how many it needed. "nextpnr" is the wall time of the whole seed-search step.Red (4)
no clean seedon 1-8no clean seedon 1-8no clean seedon 1-8; Yosys also loggedyosys-abc: abc_global.h:312: Abc_LitIsCompl: Assertion 'Lit >= 0' failed(return code 134, logged as a Warning) and carried onembedding_lookup.v:57: ERROR: Can not open file 'embedding_weights.mem' for $readmembMain's seed search does not print nextpnr's log for a failed seed (#857 is open for that), so for the three P&R reds the reason per seed is not in these runs. That is what the experiment below is for.
fpga-hslm is not a placer problem.
6f6840d52(2026-03-18, a 15,745-file deletion) removedfpga/openxc7-synth/embedding_weights.mem;e8fa551a8restored it byte-identically (blob668eaee9) but only asfpga/weights/embedding_weights.mem. Yosys runs infpga/openxc7-synth. The only other run of this workflow, 24625363076 (2026-04-19), also failed. Fix on branchfix/hslm-embedding-mem(15cec09d, copies the file in before synthesis), dispatched; whatever fails next is the next finding, since this workflow places with--placer sa --seed 1and no search. Note it targetsxc7a100tfgg676-1, not the AX7203.Green (27), sorted by how close to red
Two of the green ones stopped on their last seed. One of them, gf16-conformance, quotes Fmax, so by the rule above its placer stays; what it needs is a decision, not a sweep.
One dispatch is one sample. A workflow that took seed 1 today can take seed 9 tomorrow, which is exactly how gf10-sub went from green to red. This table says where the margin is thin, not that the rest are safe.
Step 2, started
Branch
exp/sa-audit-ab(ec7aa3b5), run 37213406352: gf12-clean, gf14-add and gf14-sub, each under--placer saand--placer heap, seeds 1-16, pinned imagesha256:35c91073, first error line of every failed seed, plus whether Yosys hit the ABC assertion. Same method as #856 round three. I will post the table here; nothing is switched before it.Side finding, for the record
gh run list --branch main --event workflow_dispatchreturned none of these 31 runs whilegh run list --workflow <file>and--createdreturned all of them. If you script against run lists, filter client-side.gHashTag commentedon Oct 4, 2026
Step 2 measured: the three red SA designs place clean under HeAP, 16 of 16
A/B on
exp/sa-audit-ab, run 37213406352. Each design ran seeds 1-16 under--placer saand--placer heapin separate jobs, with the Yosys command and the nextpnr flags copied from its workflow on main (--freq 50.0 --router router1 --timing-allow-fail), onregymm/openxc7@sha256:35c91073. "Clean" uses the gate's own test: noFailed to find a routein the log and the.fasmexists.post-placement validity check failed for Bel 'SLICE_*/A5FF' (no cell)$auto$ff.cc:337:slice$9028, 1xFailed to find a route for arc 60 of net $PACKER_VCC_NETFailed to find a route for arc 31 of net $PACKER_VCC_NETSum of nextpnr wall time over 16 seeds, SA vs HeAP: gf12_clean 170 s vs 108 s, gf14_add 340 s vs 74 s, gf14_sub 202 s vs 117 s.
Controls.
--placer saproduces invalid placements on xc7 (all revisions tested since 2026-07-28) openXC7/nextpnr-xilinx#162, the same one gf10-sub hit (ci(fpga): seed searches hide nextpnr's log when every seed fails #856, ci(fpga): place gf10-sub with HeAP, not SA #861).What this does not show.
--timing-allow-fail, and none of them records Fmax, so this does not change what they claim. On gf14_add, where SA did place, SA reached 25.56-27.93 MHz; HeAP is higher on every seed.yosys-abc ... Abc_LitIsCompl: Assertion 'Lit >= 0' failed(return code 134) as a warning under both placers. HeAP places the resulting netlist; whether that netlist is functionally right is open (S27).Next.
--placer heap. Check runs dispatched on its branch: gf12-clean 37214979643, gf14-add 37214983413, gf14-sub 37214986763.fix/hslm-embedding-mempassed Yosys in run 37213637322 (it failed there before); nextpnr was still running when this was written.ci(exp): add gf8_mul to the round-two A/B as the heap counter-example
gHashTag commentedon Oct 4, 2026
Round 2 of the placer A/B is complete: run 37215621928 (
exp-sa-audit-ab2), 5 designs × {sa, heap} × seeds 1-16, router1,--freq 50, regymm/openxc7sha256:35c91073. The MUL designs use-nodspnetlists, so the DSP confound from round 1 is gone. Each design's sources, xdc and Yosys flags match its CI workflow.Summarised with
speed ab --md:$PACKER_VCC_NETunrouted, 1xrouter1.cc:331assert$PACKER_VCC_NETunrouted$PACKER_VCC_NETunrouted, 1xrouter1.cc:331assertrouter1.cc:331assert, 1x$PACKER_VCC_NETunroutedA5FF is
post-placement validity check failed for Bel 'SLICE_.../A5FF', openXC7/nextpnr-xilinx#162. No Yosysabcassertion in any job.Totals: SA 17 of 80 clean, HeAP 80 of 80. HeAP's Fmax range is above SA's on every design, with no overlap.
What the CI loop spends today (seeds 1, 2, … until the first clean one; sum of the measured per-seed times):
Acted on: #864 switches the first four to HeAP. It also corrects the three MUL step names, which said "placer heap" while passing
--placer sa.Not acted on, owner's call:
ax7203-gf16-conformance.yml. Its loop is one seed from red: seed 8 is the last it tries and the first SA places. It prints Fmax to the log, and HeAP would move that from ~84-95 to ~258-310 MHz.Self-check that caught an anomaly before posting: the first summary showed 5 of the 10 jobs as "no seed lines". The logs were fine. Those five were fetched with
gh run view --job --log, which putsjob<TAB>step<TAB>in front of the timestamp, and the parser only knew thegh api …/jobs/ID/logsshape. The parser now accepts both and has a fixture for the second.Not measured
🤖 Generated with Claude Code
gHashTag commentedon Oct 4, 2026
S27: gf14_sub's netlist after the
yosys-abcassertion is equivalent to its input, and costs LUTsAn earlier comment left this open: gf14_sub's Yosys logs
Abc_LitIsCompl: Assertion 'Lit >= 0' failed(rc 134), and Yosys carries on. Is the netlist it carries on with correct?The run was 37230334955 on
exp/abc-equiv(15bc5b8). Toolchain:regymm/openxc7@sha256:35c91073, Yosys 0.68 (38e001a6f), yosys-abc built 2026-10-01. Flow:synth_xilinx -flatten -abc9 -nocarry -arch xc7, as in the CI workflows.What reproduces
Splitting
synth_xilinxatmap_luts(writing the pre-map netlist, then runningmap_luts:onward) gives JSON byte-identical to the one-shot run for all three designs. The gf14_sub split still asserts. The crash is deterministic, and it happens inside the post-map half.Is the post-map netlist equivalent to the pre-map one?
The method is a SAT miter with the flip-flops cut:
STARTUPE2is everted to ports;cells_sim.v;miter -equivandsat -prove-asserts -enable_undef -set-def-inputs.The cut found 154 of 154 flip-flops on both sides, with 0 problems.
no model found: SUCCESS!, 6.4 su_add.result_packed[7]invertedresult_packed[7]invertedRESULT subject_equivalent=true controls_ok=true crash_in_split=trueSo the assertion does not make the netlist wrong. From the pre-map netlist to the post-map netlist, gf14_sub is proved equivalent. The cost is area.
What the crash costs
Starting from the same written pre-map netlist (
read_rtlil+hierarchy, thenmap_luts:withabc9.nocleanup):$lutcells&mfs)&mfsremoved viaabc9.scriptOn 0.68, from the same input,
&mfsremoves 65 LCs (447 → 382, 14.5%). Locally, on Yosys 0.67 (whose abc does not assert), it removes 55 (435 → 380, 12.6%). gf14_sub as CI builds it carries 488 LCs, 106 more than 382.Corrections
yosys-abcalone on the four kept files (input.xaig,input.lut,input.box, the script) asserts. That reproducer changed two things at once: it re-reads the netlist, andabc9.nocleanupmoves the temp directory. So which of the two hides the crash is not known. A minimal report to the abc maintainers needs the crashing invocation's own input files, and those are not in hand yet.&ifmapping with&mfsskipped" is not supported. That was my first reading of the log. The no-&mfsrun gives 447, not 488. The crashed flow and the no-&mfsflow differ in more than&mfs.abc_ops_reintegratecell names that appear in the gf14_sub post-map netlist also appear in gf14_add (448 vs 270). They are not a marker of the crash.Not measured
map_lutsonward, not the passes before it.Next: one more run with
abc9.nocleanupset inside the one-process flow. If it still asserts, the kept files are the crashing input, andyosys-abcalone on them is a minimal reproducer.🤖 Generated with Claude Code
ci(exp): check the &mfs assertion against current abc
ci(exp): test whether abc a62edac0 alone fixes the &mfs assertion
gHashTag commentedon Oct 4, 2026
S27: the
&mfsassertion is already fixed upstream, and two corrections to my last commentStandalone reproducer
Run 37233120147 set
abc9.nocleanupinside the one-process CI flow. gf14_sub still asserted, andyosys-abcalone on the four kept files asserts too: rc 134,Abc_LitIsCompl: Assertion 'Lit >= 0' failedatabc_global.h:312. The files are inresearch/abc-repro/onexp/abc-equiv(input.xaigsha2567feeb347…).The crash needs these five steps after
read_lut,read_boxand&read:abc exits 0 when a script stops on a command error. So a control below counts as having ended normally only if its closing
&ps -lprintedLUT =.&sweep&dc2&dch -f -r&ifwithout-W 300&mfs&scorradded (abc9's script runs it before&sweep)read_lut(abc's default K=4 library)&if -K 6 -W 300&if -K 7 -W 300read_boxVec_IntEntryi >= 0 && i < p->nSize(vecInt.h:446)So the crash needs the 8-input LUT library Yosys writes for 7-series (
$__ABC9_LUT7and$__ABC9_LUT8) together with 8-input cuts.Current abc does not assert
Run 37234749853 built three abcs from source and ran the reproducer plus five controls on each. The input hashes matched the kept files in all three jobs.
2a9e9c2ayosys-experimental1293962a(2026-09-30)masterab2139ee(2026-09-25)Yosys
mainalready pins abc1293962a.The fix is berkeley-abc/abc#567 (
a62edac0, merged 2026-09-20): "&mfs: Don't take black-box-driven internal nodes for CI buffers". It changes six lines ofsrc/aig/gia/giaMfs.c, and its message quotes this exact assertion. The code it changes isassert( Abc_LitIsCompl(iGroup) )for a node driven by a black-box output alone, and at the v0.68 pin that line is line 428.Run 37235926503 measures it. It built the v0.68 pin twice, once as is and once with
a62edac0's diff applied, so the two builds differ only in those six lines ofgiaMfs.c(applied=true, 1 file, +4 −2). The input hashes matched the kept files in both jobs.&mfs/&sweep/&dc2/&dch/-W)2a9e9c2aa62edac0So
a62edac0alone removes the assertion, and it changes none of the five control counts. With it the reproducer maps to 504 LUTs, against 497 on current master; the other 7 come from commits between the pin and master that I have not identified.So there is nothing to report upstream. I had drafted an issue for berkeley-abc. It is retired, never sent. I should have built current master before minimising the crash over two runs: it was the cheapest control and I ran it last.
What this means here
The crash is not ours to fix. It goes away when the openxc7 image ships a Yosys whose abc includes
a62edac0. Yosys v0.68 does not; Yosysmaindoes. This is an inference from the pins, not a measurement of a release. Until then, gf14_sub keeps the pre-&mfsmapping (below), and the proof in my last comment shows that mapping is correct.Corrections to my last comment
&mfson that input, and only on that input.&ifmapping with&mfsskipped" was right, and my "correction" of it was wrong. abc9's script writes&write -n output.aigbefore&mfs. When&mfsaborts, Yosys printsWarning: ABC: execution of command ... failed: return code 134.and reads that file back: 563$lutcells, 488 LCs. The no-&mfsrun I compared it with (534$lut, 447 LCs) started from a re-read netlist, so the input differed, not just the script.Yosys warns, but its own exit code stays 0, and the flow carries on to place and route: the HeAP Fmax figures above were measured on this netlist. A step that greps the Yosys log for
failed: return codewould turn this into a visible failure. Not done here, because the netlist is proved correct and the fix is upstream.Not measured
&ps -lcounts on the kept input, not Yosys's$lutcount and not LCs after packing.Vec_IntEntryabort withoutread_boxis the same defect.a62edac0, and when the openxc7 image picks it up.🤖 Generated with Claude Code
gHashTag commentedon Oct 5, 2026
Post-merge check on main, 2026-10-05 (runs created after 18:39Z on 10-04; these workflows have not fired since).
4fc3f81aacb3de9309Negative control on the same branch: at
fe2c3ab27, the main commit between the two merges, gf12-clean (37225332454), gf14-add (37225332226) and gf14-sub (37225332104) each failed withno clean seedunder--placer sa(read from the three logs), as in the step-1 audit. The other 15 AX7203 GF conformance runs in that push atfe2c3ab27succeeded: the four #864 designs a second time, and 11 others whose placer I did not re-check. Only the three switched workflows ran atcb3de9309. (Corrected: the first version said 14, green at both commits.)Left open, not mine to decide: gf16-conformance (SA 8/8 but stops on seed 8 of 8; owner's call) and fpga-hslm (needs #866). The remaining 24 SA workflows were green in the step-1 audit and were not switched.
Not measured: seed spread on main (each run stops at its first clean seed), and whether any of the 24 SA workflows has turned fragile since 10-04.
🤖 Generated with Claude Code
ci(exp): SA/HeAP A/B for the four thin-margin --placer sa workflows
ci(exp): add gf8_mul to the round-two A/B as the heap counter-example
ci(exp): prove whether the yosys-abc crash changes the gf14-sub netlist
ci(exp): check the &mfs assertion against current abc
ci(exp): test whether abc a62edac0 alone fixes the &mfs assertion
ci(exp): fpga-hslm with --placer heap, the only change from the fix b…
ci(exp): router1 vs router2 under HeAP on the #862 conformance designs
ci(exp): SA/HeAP A/B for the three red --placer sa conformance workflows