Skip to content

ci(fpga): audit the 31 workflows that pass --placer sa against today's image #862

Open

Description

@gHashTag

Part of #851. Follows #856.

ax7203-gf10-sub was green in July and red on 2026-10-04 with no change to its inputs. The cause was the --placer sa bug that openXC7/nextpnr-xilinx#162 documents. On this design HeAP placed 24 of 24 seeds, and SA placed 11 of 40 (#856, #861).

31 other workflows on main pass --placer sa, and 29 of them have not run on main since at least 2026-08-05. The oldest of the last 1000 main runs is from that date. They run only when their own paths change. Their state on today's regymm/openxc7:latest (sha256:35c91073) is unknown, which is exactly the state gf10-sub was in.

Step 1: measure (started 2026-10-04 15:17 UTC)

Each of the 31 has been dispatched once on main. All 31 run with no side effects: an artifact upload only, with no push, release, write permission or secret. The results go in the table below.

workflow kind seed search last main run before this today
ax7203-gf10-add conformance 8 success 2026-08-05
ax7203-gf14-add conformance 8 success 2026-08-05
ax7203-gf{4,6,8,12,14,16,24}-sub conformance 8 none since 2026-08-05
ax7203-gf{4,6,8,12,16,24}-mul-clean conformance 8 none since 2026-08-05
ax7203-gf{4,6,8,12,16,24}-clean conformance 8-16, or another loop form none since 2026-08-05
ax7203-corona-decode conformance other loop form none since 2026-08-05
ax7203-identity-echo design check 16 none since 2026-08-05
ax7203-blinky-bitstream, ax7203-led-onehot, fpga-hslm-bitstream anchor, mentions Fmax fixed seed 1 none since 2026-08-05
ax7203-gf16-conformance, ax7203-dual-clk-heartbeat, ax7203-rx-echo-hb, ax7203-uart-loopback, ax7203-uart-tx-probe mentions Fmax 8 none since 2026-08-05

"kind" comes from the workflow name and from whether the file mentions Fmax. That is a first cut, not a reading of each file.

Step 2: act per workflow, not by sweep

Not in scope

  • The 6 workflows that already pass --placer heap and the 23 that pass no --placer (default heap in a WITH_HEAP build, arch.cc:1390-1394 at 1743d0f4).
  • The floating :latest tag, which affects every workflow.

Activity

  1. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    Step 1 measured: 27 green, 4 red

    All 31 dispatches on main (38da24d3a, image regymm/openxc7:latest = sha256:35c91073) ran 15:12-15:32 UTC. "Seed" is the seed the search stopped on, i.e. how many it needed. "nextpnr" is the wall time of the whole seed-search step.

    Red (4)

    workflow run failed step what the log says
    ax7203-gf12-clean 37212183150 nextpnr seed search no clean seed on 1-8
    ax7203-gf14-add 37212211972 nextpnr seed search no clean seed on 1-8
    ax7203-gf14-sub 37212248879 nextpnr seed search no clean seed on 1-8; Yosys also logged yosys-abc: abc_global.h:312: Abc_LitIsCompl: Assertion 'Lit >= 0' failed (return code 134, logged as a Warning) and carried on
    fpga-hslm-bitstream 37212434310 Yosys embedding_lookup.v:57: ERROR: Can not open file 'embedding_weights.mem' for $readmemb

    Main's seed search does not print nextpnr's log for a failed seed (#857 is open for that), so for the three P&R reds the reason per seed is not in these runs. That is what the experiment below is for.

    fpga-hslm is not a placer problem. 6f6840d52 (2026-03-18, a 15,745-file deletion) removed fpga/openxc7-synth/embedding_weights.mem; e8fa551a8 restored it byte-identically (blob 668eaee9) but only as fpga/weights/embedding_weights.mem. Yosys runs in fpga/openxc7-synth. The only other run of this workflow, 24625363076 (2026-04-19), also failed. Fix on branch fix/hslm-embedding-mem (15cec09d, copies the file in before synthesis), dispatched; whatever fails next is the next finding, since this workflow places with --placer sa --seed 1 and no search. Note it targets xc7a100tfgg676-1, not the AX7203.

    Green (27), sorted by how close to red

    workflow seed / of nextpnr Fmax line (last)
    ax7203-gf6-mul-clean 8 / 8 181 s
    ax7203-gf16-conformance 8 / 8 53 s clk200 95.35 MHz, PASS at 50
    ax7203-gf16-mul-clean 6 / 8 101 s
    ax7203-gf24-sub 6 / 8 155 s
    ax7203-dual-clk-heartbeat 5 / 8 192 s mclk_200 109.40 MHz, PASS at 50
    ax7203-gf12-sub 4 / 8 99 s
    ax7203-gf8-mul-clean 4 / 8 92 s
    ax7203-gf4-mul-clean 3 / 8 62 s
    ax7203-gf4-sub 3 / 8 68 s
    ax7203-gf8-sub 3 / 8 90 s
    ax7203-gf24-mul-clean 3 / 8 122 s
    ax7203-uart-loopback 3 / 8 28 s clk200 78.09 MHz, PASS at 50
    ax7203-gf10-add 2 / 8 74 s
    ax7203-gf16-clean 2 / 16 119 s
    ax7203-gf24-clean 2 / 16 79 s
    ax7203-corona-decode 1 / 8 32 s
    ax7203-gf12-mul-clean 1 / 8 41 s
    ax7203-gf16-sub 1 / 8 42 s
    ax7203-gf6-clean 1 / 8 32 s
    ax7203-gf6-sub 1 / 8 33 s
    ax7203-gf8-clean 1 / 8 35 s
    ax7203-gf4-clean 1 / 16 31 s
    ax7203-identity-echo 1 / 16 27 s
    ax7203-rx-echo-hb 1 / 8 35 s mclk 78.30 MHz, PASS at 50
    ax7203-uart-tx-probe 1 / 8 29 s mclk 103.26 MHz, PASS at 50
    ax7203-led-onehot fixed 1 29 s mclk 103.16 MHz, PASS at 50
    ax7203-blinky-bitstream fixed 1 13 s clk200 87.60 MHz, FAIL at 100 (allowed; it is an anchor)

    Two of the green ones stopped on their last seed. One of them, gf16-conformance, quotes Fmax, so by the rule above its placer stays; what it needs is a decision, not a sweep.

    One dispatch is one sample. A workflow that took seed 1 today can take seed 9 tomorrow, which is exactly how gf10-sub went from green to red. This table says where the margin is thin, not that the rest are safe.

    Step 2, started

    Branch exp/sa-audit-ab (ec7aa3b5), run 37213406352: gf12-clean, gf14-add and gf14-sub, each under --placer sa and --placer heap, seeds 1-16, pinned image sha256:35c91073, first error line of every failed seed, plus whether Yosys hit the ABC assertion. Same method as #856 round three. I will post the table here; nothing is switched before it.

    Side finding, for the record

    gh run list --branch main --event workflow_dispatch returned none of these 31 runs while gh run list --workflow <file> and --created returned all of them. If you script against run lists, filter client-side.

  2. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    Step 2 measured: the three red SA designs place clean under HeAP, 16 of 16

    A/B on exp/sa-audit-ab, run 37213406352. Each design ran seeds 1-16 under --placer sa and --placer heap in separate jobs, with the Yosys command and the nextpnr flags copied from its workflow on main (--freq 50.0 --router router1 --timing-allow-fail), on regymm/openxc7@sha256:35c91073. "Clean" uses the gate's own test: no Failed to find a route in the log and the .fasm exists.

    design SA clean SA s/seed why SA failed HeAP clean HeAP s/seed HeAP Fmax (clean seeds)
    gf12_clean 0 of 16 10-12 16x post-placement validity check failed for Bel 'SLICE_*/A5FF' (no cell) 16 of 16 6-8 48.74-52.97 MHz
    gf14_add 3 of 16 (seeds 10, 11, 15) 11-60; clean ones 44-52 11x A5FF (no cell), 1x A5FF holding $auto$ff.cc:337:slice$9028, 1x Failed to find a route for arc 60 of net $PACKER_VCC_NET 16 of 16 4-5 43.54-47.87 MHz
    gf14_sub 0 of 16 10-36 15x A5FF (no cell), 1x Failed to find a route for arc 31 of net $PACKER_VCC_NET 16 of 16 6-9 36.78-39.86 MHz

    Sum of nextpnr wall time over 16 seeds, SA vs HeAP: gf12_clean 170 s vs 108 s, gf14_add 340 s vs 74 s, gf14_sub 202 s vs 117 s.

    Controls.

    What this does not show.

    • HeAP's Fmax on gf14_add and gf14_sub is below the 50 MHz target. These are conformance workflows with --timing-allow-fail, and none of them records Fmax, so this does not change what they claim. On gf14_add, where SA did place, SA reached 25.56-27.93 MHz; HeAP is higher on every seed.
    • gf14_sub's Yosys logs yosys-abc ... Abc_LitIsCompl: Assertion 'Lit >= 0' failed (return code 134) as a warning under both placers. HeAP places the resulting netlist; whether that netlist is functionally right is open (S27).
    • One run per job, one image. No board load.

    Next.

  3. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    Round 2 of the placer A/B is complete: run 37215621928 (exp-sa-audit-ab2), 5 designs × {sa, heap} × seeds 1-16, router1, --freq 50, regymm/openxc7 sha256:35c91073. The MUL designs use -nodsp netlists, so the DSP confound from round 1 is gone. Each design's sources, xdc and Yosys flags match its CI workflow.

    Summarised with speed ab --md:

    design placer clean s/seed Fmax of clean seeds why the others failed
    gf16_mul sa 4 of 16 10-62 27.86-37.2 MHz 12x A5FF (no cell)
    gf16_mul heap 16 of 16 8-10 60.59-66.48 MHz -
    gf8_mul sa 4 of 16 6-38 42.27-51.41 MHz 10x A5FF (no cell), 1x $PACKER_VCC_NET unrouted, 1x router1.cc:331 assert
    gf8_mul heap 16 of 16 5-7 102.24-115.97 MHz -
    gf6_mul sa 4 of 16 6-108 42.43-59.12 MHz 7x A5FF (no cell), 1x A5FF naming a cell, 4x $PACKER_VCC_NET unrouted
    gf6_mul heap 16 of 16 5-6 146.58-159.21 MHz -
    gf16_codec sa 2 of 16 3-70 83.65-95.35 MHz 11x A5FF (no cell), 1x A5FF naming a cell, 1x $PACKER_VCC_NET unrouted, 1x router1.cc:331 assert
    gf16_codec heap 16 of 16 2-4 258.33-309.79 MHz -
    gf24_sub sa 3 of 16 21-72 15.55-16.17 MHz 11x A5FF (no cell), 1x router1.cc:331 assert, 1x $PACKER_VCC_NET unrouted
    gf24_sub heap 16 of 16 6-9 26.33-28.12 MHz -

    A5FF is post-placement validity check failed for Bel 'SLICE_.../A5FF', openXC7/nextpnr-xilinx#162. No Yosys abc assertion in any job.

    Totals: SA 17 of 80 clean, HeAP 80 of 80. HeAP's Fmax range is above SA's on every design, with no overlap.

    What the CI loop spends today (seeds 1, 2, … until the first clean one; sum of the measured per-seed times):

    workflow SA, first clean seed SA step HeAP step
    gf16-mul-clean 6 98 s 10 s
    gf8-mul-clean 4 80 s 6 s
    gf6-mul-clean 8 194 s 6 s
    gf24-sub 6 215 s 9 s
    gf16-conformance 8 63 s 4 s

    Acted on: #864 switches the first four to HeAP. It also corrects the three MUL step names, which said "placer heap" while passing --placer sa.

    Not acted on, owner's call: ax7203-gf16-conformance.yml. Its loop is one seed from red: seed 8 is the last it tries and the first SA places. It prints Fmax to the log, and HeAP would move that from ~84-95 to ~258-310 MHz.

    Self-check that caught an anomaly before posting: the first summary showed 5 of the 10 jobs as "no seed lines". The logs were fine. Those five were fetched with gh run view --job --log, which puts job<TAB>step<TAB> in front of the timestamp, and the parser only knew the gh api …/jobs/ID/logs shape. The parser now accepts both and has a fixture for the second.

    Not measured

    🤖 Generated with Claude Code

  4. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    S27: gf14_sub's netlist after the yosys-abc assertion is equivalent to its input, and costs LUTs

    An earlier comment left this open: gf14_sub's Yosys logs Abc_LitIsCompl: Assertion 'Lit >= 0' failed (rc 134), and Yosys carries on. Is the netlist it carries on with correct?

    The run was 37230334955 on exp/abc-equiv (15bc5b8). Toolchain: regymm/openxc7@sha256:35c91073, Yosys 0.68 (38e001a6f), yosys-abc built 2026-10-01. Flow: synth_xilinx -flatten -abc9 -nocarry -arch xc7, as in the CI workflows.

    What reproduces

    design runs abc assertions est. LCs JSON
    gf14_sub 3 1 in each run 488 identical in all 3 runs
    gf14_add 1 0 380
    gf12_clean 1 0 352

    Splitting synth_xilinx at map_luts (writing the pre-map netlist, then running map_luts: onward) gives JSON byte-identical to the one-shot run for all three designs. The gf14_sub split still asserts. The crash is deterministic, and it happens inside the post-map half.

    Is the post-map netlist equivalent to the pre-map one?

    The method is a SAT miter with the flip-flops cut:

    • each of the 154 FDCE/FDPE becomes a free input for its Q plus outputs for its D, its reset and its clock;
    • STARTUPE2 is everted to ports;
    • both sides are techmapped to cells_sim.v;
    • then miter -equiv and sat -prove-asserts -enable_undef -set-def-inputs.

    The cut found 154 of 154 flip-flops on both sides, with 0 problems.

    case top expected result
    subject gf14_sub (asserted) equal proved, no model found: SUCCESS!, 6.4 s
    positive gf12_clean equal proved
    positive gf14_add equal proved
    negative: first LUT6 inverted gf14_sub differ counterexample, 1 of 473 outputs
    negative: LUT6 driving u_add.result_packed[7] inverted gf14_sub differ counterexample, 1 of 473
    negative: same LUT6, INIT bit 0 flipped gf14_sub differ counterexample, 1 of 473
    negative: LUT6 driving result_packed[7] inverted gf14_add differ counterexample, 1 of 473

    RESULT subject_equivalent=true controls_ok=true crash_in_split=true

    So the assertion does not make the netlist wrong. From the pre-map netlist to the post-map netlist, gf14_sub is proved equivalent. The cost is area.

    What the crash costs

    Starting from the same written pre-map netlist (read_rtlil + hierarchy, then map_luts: with abc9.nocleanup):

    run abc assertions $lut cells est. LCs
    abc9 default script (with &mfs) 0 496 382
    &mfs removed via abc9.script 0 534 447
    the CI flow, in one process (asserts) 1 — 488

    On 0.68, from the same input, &mfs removes 65 LCs (447 → 382, 14.5%). Locally, on Yosys 0.67 (whose abc does not assert), it removes 55 (435 → 380, 12.6%). gf14_sub as CI builds it carries 488 LCs, 106 more than 382.

    Corrections

    • The assertion did not reproduce from the written netlist. Neither the re-read run nor yosys-abc alone on the four kept files (input.xaig, input.lut, input.box, the script) asserts. That reproducer changed two things at once: it re-reads the netlist, and abc9.nocleanup moves the temp directory. So which of the two hides the crash is not known. A minimal report to the abc maintainers needs the crashing invocation's own input files, and those are not in hand yet.
    • "The 488-LC netlist is the &if mapping with &mfs skipped" is not supported. That was my first reading of the log. The no-&mfs run gives 447, not 488. The crashed flow and the no-&mfs flow differ in more than &mfs.
    • The abc_ops_reintegrate cell names that appear in the gf14_sub post-map netlist also appear in gf14_add (448 vs 270). They are not a marker of the crash.

    Not measured

    • Whether the pre-map netlist matches the RTL. The proof covers map_luts onward, not the passes before it.
    • Whether the 106 extra LCs change Fmax after place and route. gf14_sub's HeAP Fmax of 36.78–39.86 MHz (above) was measured on the asserted netlist only.
    • Whether a newer abc still asserts. The image's abc is the one Yosys 0.68 ships.

    Next: one more run with abc9.nocleanup set inside the one-process flow. If it still asserts, the kept files are the crashing input, and yosys-abc alone on them is a minimal reproducer.

    🤖 Generated with Claude Code

  5. gHashTag commented on Oct 4, 2026

    @gHashTag
    OwnerAuthor

    S27: the &mfs assertion is already fixed upstream, and two corrections to my last comment

    Standalone reproducer

    Run 37233120147 set abc9.nocleanup inside the one-process CI flow. gf14_sub still asserted, and yosys-abc alone on the four kept files asserts too: rc 134, Abc_LitIsCompl: Assertion 'Lit >= 0' failed at abc_global.h:312. The files are in research/abc-repro/ on exp/abc-equiv (input.xaig sha256 7feeb347…).

    The crash needs these five steps after read_lut, read_box and &read:

    &sweep
    &dc2
    &dch -f -r
    &if -W 300
    &mfs
    

    abc exits 0 when a script stops on a command error. So a control below counts as having ended normally only if its closing &ps -l printed LUT =.

    change to the script (local, Yosys 0.67+post b8e7da6f, macOS arm64) result
    none asserts
    no &sweep ends normally, 505 LUTs
    no &dc2 499
    no &dch -f -r 509
    &if without -W 300 480
    no &mfs 559
    &scorr added (abc9's script runs it before &sweep) still asserts
    no read_lut (abc's default K=4 library) ends normally, 577
    &if -K 6 -W 300 ends normally, 463
    &if -K 7 -W 300 ends normally, 436
    no read_box a different assertion: Vec_IntEntry i >= 0 && i < p->nSize (vecInt.h:446)

    So the crash needs the 8-input LUT library Yosys writes for 7-series ($__ABC9_LUT7 and $__ABC9_LUT8) together with 8-input cuts.

    Current abc does not assert

    Run 37234749853 built three abcs from source and ran the reproducer plus five controls on each. The input hashes matched the kept files in all three jobs.

    abc commit reproducer controls
    YosysHQ/abc, the commit Yosys v0.68 pins 2a9e9c2a asserts, rc 134 (positive control) 5/5 end normally, with the same LUT counts as the local table
    YosysHQ/abc yosys-experimental 1293962a (2026-09-30) ends normally, 497 LUTs 5/5
    berkeley-abc/abc master ab2139ee (2026-09-25) ends normally, 497 LUTs 5/5

    Yosys main already pins abc 1293962a.

    The fix is berkeley-abc/abc#567 (a62edac0, merged 2026-09-20): "&mfs: Don't take black-box-driven internal nodes for CI buffers". It changes six lines of src/aig/gia/giaMfs.c, and its message quotes this exact assertion. The code it changes is assert( Abc_LitIsCompl(iGroup) ) for a node driven by a black-box output alone, and at the v0.68 pin that line is line 428.

    Run 37235926503 measures it. It built the v0.68 pin twice, once as is and once with a62edac0's diff applied, so the two builds differ only in those six lines of giaMfs.c (applied=true, 1 file, +4 −2). The input hashes matched the kept files in both jobs.

    build reproducer the five controls (no &mfs / &sweep / &dc2 / &dch / -W)
    pin 2a9e9c2a asserts, rc 134, same message 559 / 505 / 499 / 509 / 480
    pin + a62edac0 ends normally, 504 LUTs 559 / 505 / 499 / 509 / 480

    So a62edac0 alone removes the assertion, and it changes none of the five control counts. With it the reproducer maps to 504 LUTs, against 497 on current master; the other 7 come from commits between the pin and master that I have not identified.

    So there is nothing to report upstream. I had drafted an issue for berkeley-abc. It is retired, never sent. I should have built current master before minimising the crash over two runs: it was the cheapest control and I ran it last.

    What this means here

    The crash is not ours to fix. It goes away when the openxc7 image ships a Yosys whose abc includes a62edac0. Yosys v0.68 does not; Yosys main does. This is an inference from the pins, not a measurement of a release. Until then, gf14_sub keeps the pre-&mfs mapping (below), and the proof in my last comment shows that mapping is correct.

    Corrections to my last comment

    1. "Locally, on Yosys 0.67 (whose abc does not assert)" was wrong. The local 0.67+post abc asserts on the kept input, the same as 0.68's. The local run that did not assert used a re-read netlist, so its input to abc was different. The 435 → 380 LC figure from that run stays valid as a measurement of &mfs on that input, and only on that input.
    2. "The 488-LC netlist is the &if mapping with &mfs skipped" was right, and my "correction" of it was wrong. abc9's script writes &write -n output.aig before &mfs. When &mfs aborts, Yosys prints Warning: ABC: execution of command ... failed: return code 134. and reads that file back: 563 $lut cells, 488 LCs. The no-&mfs run I compared it with (534 $lut, 447 LCs) started from a re-read netlist, so the input differed, not just the script.

    Yosys warns, but its own exit code stays 0, and the flow carries on to place and route: the HeAP Fmax figures above were measured on this netlist. A step that greps the Yosys log for failed: return code would turn this into a visible failure. Not done here, because the netlist is proved correct and the fix is upstream.

    Not measured

    • gf14_sub's LC count and Fmax with a fixed abc in the real flow. The 497- and 504-LUT figures are abc's own &ps -l counts on the kept input, not Yosys's $lut count and not LCs after packing.
    • Which commits between the pin and master account for 504 → 497.
    • Whether the Vec_IntEntry abort without read_box is the same defect.
    • Which Yosys release first ships a62edac0, and when the openxc7 image picks it up.

    🤖 Generated with Claude Code

  6. gHashTag commented on Oct 5, 2026

    @gHashTag
    OwnerAuthor

    Post-merge check on main, 2026-10-05 (runs created after 18:39Z on 10-04; these workflows have not fired since).

    merge workflows switched to HeAP first main run result
    #864 4fc3f81aa gf6-mul-clean, gf8-mul-clean, gf16-mul-clean, gf24-sub 37225274843, 37225274964, 37225274573, 37225274984 4/4 success
    #863 cb3de9309 gf12-clean, gf14-add, gf14-sub 37225379245, 37225379150, 37225379219 3/3 success

    Negative control on the same branch: at fe2c3ab27, the main commit between the two merges, gf12-clean (37225332454), gf14-add (37225332226) and gf14-sub (37225332104) each failed with no clean seed under --placer sa (read from the three logs), as in the step-1 audit. The other 15 AX7203 GF conformance runs in that push at fe2c3ab27 succeeded: the four #864 designs a second time, and 11 others whose placer I did not re-check. Only the three switched workflows ran at cb3de9309. (Corrected: the first version said 14, green at both commits.)

    Left open, not mine to decide: gf16-conformance (SA 8/8 but stops on seed 8 of 8; owner's call) and fpga-hslm (needs #866). The remaining 24 SA workflows were green in the step-1 audit and were not switched.

    Not measured: seed spread on main (each run stops at its first clean seed), and whether any of the 24 SA workflows has turned fragile since 10-04.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions