Skip to content

fix(ci): key the chipdb cache on the openXC7 image so it can hit - #852

Open
gHashTag wants to merge 2 commits into
mainfrom
fix/ci-chipdb-cache
Open

gHashTag wants to merge 2 commits into
mainfrom
fix/ci-chipdb-cache

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner

Part of #851. Fixes #843 and #845.

What was wrong

61 workflows build the nextpnr-xilinx chipdb for an openXC7 job. Before this PR only one of them could reuse it:

class before after
OK: an exact key that changes when the image changes 1 54
NEVER_HIT: key carries github.run_id, so cache-hit is never true and the build step always runs (#843) 48 0
STALE_RISK: fixed key on a floating regymm/openxc7 tag (#845) 5 0
NO_CACHE: other parts or a source-built nextpnr (#846) 7 7

Counts are from tri devkit speed coverage --repo ., run on main and on this branch.

What changes

A composite action, .github/actions/openxc7-chipdb:

  1. It resolves the image the job's nextpnr runs to its local image ID. It pulls the image only if the runner does not already have it, so the chipdb and the job use the same image.
  2. It restores ~/.cache/openxc7-chipdb/<part>.bin under the key chipdb-<part>-<image id>-bbasm-le-v1.
  3. On an exact miss it builds the chipdb inside that image (bbaexport.py then bbasm -l). It saves it at once with actions/cache/save, so a later nextpnr failure does not lose it.
  4. It copies the file to the workflow's own out path.

Why one cached path for every workflow. actions/cache folds the path list into the entry version. Caching each workflow's out path directly would store 50 copies of one identical file.

53 workflows have their cache step and guarded build step replaced by one step:

- id: chipdb
  uses: ./.github/actions/openxc7-chipdb
  with:
    part: xc7a200tfbg484-2
    out: build/gf10_sub/chipdb/xc7a200tfbg484-2.bin

The step keeps the id chipdb, so any later steps.chipdb.outputs.cache-hit still resolves. The four build-* workflows pass image: regymm/openxc7:latest. The two tnf-* workflows pass their pinned @sha256: image.

How the rewrite was checked

  • The rewrite was done by a script, not by hand. For each file:
    • the new YAML must parse to the old document with exactly the two chipdb steps replaced by the one above;
    • the file must re-classify as OK.
  • The script refuses shapes it does not model:
    • more than one chipdb cache step;
    • no guarded build step next to it;
    • more than one image;
    • shell variables in the part or path;
    • no actions/checkout before the step.
  • Negative controls:
    • 5 of 5 corrupted rewrites are refused: a wrong out path, a dropped step name, a changed job field, a lost id, and the old build step kept.
    • 5 of 5 mutants of the coverage checker are killed.
  • The coverage checker reads the action itself. It reports NEVER_HIT if the action is missing, or if its key loses the image ID, uses a per-run key, caches out directly, or does not save early.
  • All 54 changed YAML files parse. The action's run steps all name a shell.
  • shellcheck is clean on the action's three run steps.
  • actionlint (with shellcheck) on the changed workflows reports 124 findings. All of them are shellcheck notes on run steps this PR does not touch, and none mentions the chipdb step.

When this merges

46 of the 53 changed workflows list their own file in push: paths, so the merge push starts 46 FPGA runs at once.

  • They run in parallel against an empty main cache, so each one builds its chipdb, as every run does today. One of them saves the entry, and the rest log that another job is creating it.
  • From then on, every run on main, and every branch and PR, restores that entry while the image stays the same.
  • If ci(cache): prune Actions cache entries nothing can restore #853 merges first, the entry also has room to survive.

Measured on CI

ax7203-gf10-sub.yml was run twice on this branch, with main as the control.

run ref chipdb step result
37197232296 this branch, first 3 m 19 s "Cache not found", then built and saved before nextpnr (11:09:41; nextpnr starts 11:09:41)
37198736545 this branch, second 1 s Cache hit for: chipdb-xc7a200tfbg484-2-591b377b…-bbasm-le-v1, about 68 MB restored, build step skipped, 332105196 bytes cache-hit=true
37198621522 main, control 3 m 24 s built; the Post Run actions/cache@v4 step was skipped because the job failed, so nothing was saved

All three fail later, at the nextpnr seed search, with a bare no clean seed. main fails the same way, so the failure was already there. Its last success on main was 2026-07-14, before regymm/openxc7:latest moved. Why it fails is #856, and #857 makes the log visible.

That failure also shows something the "after" column of the table above does not. On main, a run that fails after the chipdb step never saves the chipdb, so every failing run pays the build again. With the explicit early save here, the first run pays once, failure or not.

A change to the action itself (0c5f415)

The action is called by 46 workflows, and none of their triggers covered .github/actions/openxc7-chipdb/. So a change to the action alone ran nothing, on the PR or on main. speed actionpaths reported 0 PR and 0 push triggers on this branch before the commit, and 1 and 1 after. ax7203-led-onehot.yml, the smallest design with one seed, now runs on a PR or a push to main that touches the action.

Its first PR run, 37260180318 on refs/pull/852/merge, succeeded: chipdb step 2 m 54 s (Cache not found, built, then saved chipdb-xc7a200tfbg484-2-591b377b…-bbasm-le-v1, cache-hit=false), nextpnr 29 s, 9.3 MB .bit. It missed because a PR run reads only its merge ref and main, and the entry saved on this branch on 2026-10-04 had already been evicted (#844 comment 5987858206; #895 is the fix).

Not measured

🤖 Generated with Claude Code

The chipdb cache in 48 openXC7 workflows was keyed on github.run_id. actions/cache
sets cache-hit only on an exact key match, so the guarded build step ran on every
run and the cache never saved a second. Five more used a fixed key on a floating
image tag, which never misses but can serve a chipdb built by an image the job no
longer runs.

A composite action, .github/actions/openxc7-chipdb, now owns this:

- key = part + the ID of the image the job's nextpnr runs (docker image inspect),
  so it hits on every run with the same image and misses exactly when it moves;
- one cached path for every workflow (~/.cache/openxc7-chipdb/<part>.bin), copied
  to each workflow's own `out`, because actions/cache folds the path into the entry
  version and 50 distinct paths would store 50 copies of one file;
- saved with actions/cache/save straight after the build, so a later nextpnr
  failure does not discard it.

The 53 workflows were rewritten by a gated script: each rewrite must parse to the
old document with exactly the two chipdb steps replaced by one, must re-classify as
OK, and is refused for any shape it does not model (more than one cache step, more
than one image, shell variables in the part or path, no checkout before it).

Coverage by the speed-loop classifier, 61 chipdb-building workflows:
before 1 OK, 48 NEVER_HIT, 5 STALE_RISK, 7 NO_CACHE; after 54 OK, 7 NO_CACHE.
The remaining 7 build other parts or their own nextpnr and are tracked in #846.

Part of #851. Fixes #843, #845.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown

X Brain Health Check

Score: 100.0/100
Status: 🟢 HEALTHY
Threshold: 80/100

X Brain is above merge threshold

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown

🧪 Brain Stress Test Results

Score: not measured (no stress test prints a Score: line)
Status: 🟢 PASS
Threshold: 270/300

X Stress test passed

The openxc7-chipdb action is called by 46 workflows, and every one of them
filters push on its own design files only, with no pull_request trigger. A
change to .github/actions/openxc7-chipdb/ alone therefore ran nothing, on the
PR or on main. Measured on this branch with a paths-coverage script: 46 of 46
triggers did not cover the action.

- ax7203-led-onehot.yml (the smallest design, one seed) now runs on a PR that
  touches the action, and on a push to main that does, so main also warms the
  cache for a new key.
- run-syntax-gate.yml watches .github/actions/**, which this PR creates; the
  gate reads composite actions once #894 lands.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gHashTag

gHashTag commented Oct 5, 2026

Copy link
Copy Markdown
Owner Author

Merge-order note: this PR and #897 both edit the chipdb steps of tnf-cost-sweep.yml, and git merge-tree reports a content conflict there. This PR swaps actions/cache + "Build chipdb" for the openxc7-chipdb composite action; #897 adds a "Package pins" step right after "Build chipdb". Resolution: keep the composite action, then the package-pins step. Whichever merges second gets rebased; I will do that after the first one lands.

🤖 Generated with Claude Code

dmitrii-f-t27 pushed a commit that referenced this pull request Oct 10, 2026
…ched

Seven workflows still built their chipdb on every run after the composite
action landed on this branch (#852). Two of them run often enough to pay
for a cache; five are dispatch-only probes where a cache would be wrong or
would change an instrument for nothing.

Cached through .github/actions/openxc7-chipdb:

- part-coverage.yml runs daily on a 4-part matrix and ran bbaexport for
  every part every time. In run 37196269484 its "Build it" step, which
  included the chipdb, took 87 s (xc7a35t), 437 s (xc7k325t), 87 s
  (xc7s50) and 351 s (xc7a200t). The chipdb step has continue-on-error, so
  a chipdb failure is still an INCONCLUSIVE verdict, as it was when the
  chipdb was built inside the build step.
- fpga-hslm-bitstream.yml, keyed on the ID of regymm/openxc7:latest, so it
  misses exactly when the tag moves.

Left uncached, each with a "# chipdb-uncached: <reason>" line above the
step that builds it:

- openxc7-vs-vivado-openxc7-half, bufr-divide-probe, iddr-vivado-diff:
  the chipdb comes from a nextpnr-xilinx revision checked out in the job,
  not from the image the cache keys on, so an image-keyed entry would be
  the wrong chipdb.
- iddr-golden-diff, router-vcc-reducer: the action would fit, but they
  have run 3 times each, last on 2026-08-18, and they are the instruments
  behind openXC7/nextpnr-xilinx#114.

The loop's classifier (speed coverage) now counts a stated reason as
decided, not as cached: on this branch 56/61 workflows cache the chipdb
and 61/61 either cache it or say why not. main is 1/61.

Not measured: whether bbasm resolves on PATH inside the pinned image
eced1cdd that part-coverage uses. That image installs it with
"cp bbasm /usr/local/bin/ || true". A dispatch of part-coverage on this
branch settles it.

Refs #846 #851

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dmitrii-f-t27 pushed a commit that referenced this pull request Oct 10, 2026
… gate

check_run_syntax.py read .github/workflows/*.yml only, so a run step inside
a composite action (.github/actions/**/action.yml, as #852 adds) could fail
to parse and the gate would pass. Closes #854.

- Composite actions are read like workflows: bash -n per shell, and the
  single-quoted bash -c scripts inside a step, via a shared check_run().
- A composite run step must name its shell; one that does not is cannot
  tell (rc 2), as GitHub itself rejects it.
- The runner is the caller's: every job that uses the action, directly or
  through other local actions (cycles end). A macOS-possible caller under
  a gate whose bash is not 3.x is cannot tell, as for workflow steps.
- An action no workflow calls is still parsed. Node and Docker actions
  have no run steps and are skipped.

Self-test 98 of 98 (23 new cases); 17 of 18 mutants of the new code are
killed, the survivor is equivalent. On #852's tree the gate reads 1 of 1
actions and catches an unclosed quote planted in its last step, which the
gate at 54020cd passes.

.github/actions/** is not added to the gate's paths: here: no tracked file
matches it on main yet, and the workflow-path gate rejects that entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dmitrii-f-t27 pushed a commit that referenced this pull request Oct 10, 2026
The note said .github/actions/ is not watched by this workflow's paths:
until a tracked file exists there. #852 creates the directory and is the
PR that adds the entry, so the note would be wrong on whichever of the two
merges second. The reason stays in #894's description.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dmitrii-f-t27 pushed a commit that referenced this pull request Oct 10, 2026
The prune from #853 keeps the newest entry of every series on every live
ref. On 2026-10-05 that alone came to 7.97 GB (130 entries), and the
saves between two prunes took the repository past 10 GB, where GitHub
evicts by last read, fixed keys included. The chipdb saved on
fix/ci-chipdb-cache on 2026-10-04 was gone by the next morning, and the
LED one-hot run on #852 rebuilt it (2 m 54 s).

Two changes:

- cache-prune.js: after the existing rules, while the kept entries are
  over BUDGET_BYTES (8e9), drop setup-zig's per-run entries on refs
  other than main, least recently read first. Nothing else is dropped
  for the budget: not a fixed key, not main, and not a chipdb whose key
  ends in a run id (one branch saves chipdb-...-regymm-<run id>). A
  chipdb hit saves about three minutes; a Zig cache hit tens of seconds
  at most.
- actions-cache-prune.yml: also run on workflow_run after brain-ci and
  CI complete. The hourly schedule started 2 runs in 9 slots between
  18:41Z and 03:46Z. brain-ci's jobs hold 8.59 GB and CI's 1.85 GB of
  10.84 GB.

Eight new tests (22 in all); 13 of 13 mutants of the budget rule are
killed. On the 162-entry list of 2026-10-05 the plan drops 43 entries
(3.06 GB), 10 of them (0.72 GB) for the budget, and keeps 119 (7.78 GB),
including all 5 chipdb entries and all 9 entries on main.

Refs #844

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@tri-claw

tri-claw Bot commented Oct 10, 2026

Copy link
Copy Markdown

Summary by t27-bees

This PR replaces per-workflow chipdb cache+build steps with a shared composite action keyed on the Docker image ID (not github.run_id), fixing cache misses on every run (#843) and stale caches from floating tags (#845). The action resolves the image to its local ID, restores/builds/saves the chipdb to a single canonical path, and copies it to each workflow's out directory.

Worth a look

  • .github/actions/openxc7-chipdb/action.yml:54 – docker pull -q "$IMAGE" >/dev/null silences pull failures; if the pull fails (network, auth, missing image), the subsequent docker image inspect fails with an obscure error and the root cause is hidden.
  • .github/actions/openxc7-chipdb/action.yml:53–55 – No set -e/set -o pipefail in the ident step; a transient docker image inspect error (e.g., daemon hiccup) would leave image_id empty, producing a malformed key and likely a confusing downstream failure.
  • .github/actions/openxc7-chipdb/action.yml:65–70 – The build step runs bbaexport.py and bbasm via bash -c with && chaining; if bbaexport.py crashes with a non-zero exit but prints to stdout/stderr, the failure is captured, but any command outside the chain (e.g., the mkdir -p before docker run) would not fail the step on error.
  • .github/actions/openxc7-chipdb/action.yml:73–77 – The actions/cache/save step runs only on cache miss. If the restore step reports a hit but the cached file is corrupted/truncated, the build is skipped and the workflow proceeds with a bad chipdb.
  • .github/workflows/ax7203-trinet-baud.yml:66 / ax7203-trinet-fleet.yml:103 – The out path uses ${{ matrix.name }}; the action’s copy step (cp -f "$HOME/.cache/openxc7-chipdb/$PART.bin" "$OUT") receives the expanded path via env.OUT, which works, but the action does not validate that OUT’s parent directory is writable or that the matrix expansion doesn’t introduce unexpected characters.

Read: 55 files, +330 -605, the whole diff up to 60000 characters (the rest was not read). Head 779168a. Model: openai-compatible/nvidia/nemotron-3-ultra-550b-a55b. Commands: @t27-bees review | summary | help | pause | resume. Free for every repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ci: chipdb cache never hits in 48 workflows (key carries github.run_id)

1 participant