Skip to content

ci(brain): give each unit-test region its own Zig cache key - #891

Open
gHashTag wants to merge 3 commits into
mainfrom
ci/brain-unit-cache-key
Open

gHashTag wants to merge 3 commits into
mainfrom
ci/brain-unit-cache-key

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 5, 2026

Copy link
Copy Markdown
Owner

Part of #844 and #851 (W4b). Stacked on #890, so merge #890 first. Until then this diff also shows #890's two commits.

What

Sets cache-key: ${{ matrix.region }} on the mlugg/setup-zig step of brain-ci's brain-unit job. That job has five matrix legs. Nothing else changes.

Why

setup-zig's key is setup-zig-cache-v2-<job id>-<tarball>-<cache-key>-<run id>-<attempt>. With cache-key empty, all five legs of brain-unit get the same key. The first leg to finish saves, and the other four fail with "Unable to reserve cache". Over the last 25 runs on main, 88 of 110 saves were refused.

In the next run every leg restores that one region's build, which is useless for the other four. Each leg tests a different region, so they compile their own test binary from nothing. setup-zig's action.yml documents cache-key for exactly this case: "When using a matrix strategy, this should include the matrix variables to ensure all jobs are cached."

Measured

I measured this with two dispatches of this branch at 9256591. That commit has the same brain-ci.yml as this head, which has since been rebased onto #890's new head. The figures come from speed zigcache --workflow brain-ci.yml --branch ci/brain-unit-cache-key --runs 2.

  • Gen 1, run 37251590313. All five legs started cold, since the keys were new, and all five saves succeeded. None was refused. On main about 4 of 5 are refused.
  • Gen 2, run 37252199768. All five legs hit the entry saved under their own region, for example …-0.16.0-hslm-37251590313-1.
leg gen 1 (cold) gen 2 (own entry) .zig-cache GB
locus-coeruleus 11.2 s 0.6 s 0.11 → 0.11
reticular-formation 19.5 s 8.8 s 0.11 → 0.11
basal-ganglia 20.2 s 18.8 s 0.11 → 0.17
hslm 7.0 s 7.6 s 0.10 → 0.16
intraparietal 4.9 s 7.7 s 0.10 → 0.16

Two of the five legs reused their entry, since their directory did not grow. The other three each rebuilt about 60 MB.

#844 found that a warm entry is reused only on the runner CPU model that saved it, so a 2-of-5 reuse rate is what that finding predicts. That link is an inference, though. This branch has no CPU probe step, so I don't know which CPU each leg ran on. The CPU part is pinned with -Dcpu, which is measured in #844 and is a separate change for the owner to decide.

What this PR guarantees is that each leg restores its own region's build. Whether that build can be reused then depends only on the CPU model, and no longer on which leg happened to save first.

Cost

Each run now stores five entries instead of one. Gen 2 restored entries of 26.2–26.8 MB, so this is about 132 MB per run against about 26 MB. They count toward the same 10 GB Actions cache limit as the chipdb entries. #853 prunes entries that nothing can restore.

Not measured

  • Which CPU each leg ran on (no probe step on this branch).
  • More than one pair of runs.
  • How these entries interact with the 10 GB limit over a week of pushes.

🤖 Generated with Claude Code

gHashTag and others added 3 commits October 5, 2026 07:43
mlugg/setup-zig@v2 restores its cache into $GITHUB_WORKSPACE/.zig-cache.
In all eight Zig jobs of brain-ci.yml it ran before actions/checkout,
which finds no clone in the workspace and empties it ("Deleting the
contents of '/home/runner/work/trinity-fpga/trinity-fpga'"). Every
restored cache was deleted before a build step used it.

Measured with `speed zigcache` over the last 25 completed brain-ci runs:
238 restores, 0 used. 237 were hits that checkout then deleted; 1 was a
miss. Each job still paid about 1-1.8 s to restore and 1-2.4 s to save
26-71 MB. On 2026-10-05 those saves were 157 of the 179 Actions cache
entries, 7.95 GB of 11.04 GB, where they evict the chipdb (#844).

Check out first, in every Zig job. No step, condition or input changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… cache

mlugg/setup-zig restores into $GITHUB_WORKSPACE/.zig-cache, and an
actions/checkout of the workspace after it deletes the restored cache.
This is how brain-ci lost every hit (238 of 238 in 25 runs), and nothing
stops the old order from coming back.

tools/check_setup_zig_order.py reads every workflow with PyYAML and exits 1
for a caching setup-zig step followed by a checkout of the workspace. A
checkout into a subdirectory and `use-cache: false` both pass. It exits 2,
not 0, when no job uses setup-zig or a workflow does not parse.

tools/test_check_setup_zig_order.py plants 17 cases. Four mutants of the
checker are each caught. On this branch: 122 workflows, 16 setup-zig jobs,
0 findings. On main's brain-ci.yml: 8 of 8 jobs reported.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The five brain-unit matrix legs got one setup-zig key, so the first leg
to finish saved and the other four were refused ("Unable to reserve
cache"): 88 of 110 saves in 25 runs. Every leg then restored that one
region's build and compiled its own test binary from nothing. setup-zig
documents `cache-key` for exactly this: include the matrix variables.

Stacked on #890, which makes the restored cache survive checkout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

X Brain Health Check

Score: 100.0/100
Status: 🟢 HEALTHY
Threshold: 80/100

X Brain is above merge threshold

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

🧪 Brain Stress Test Results

Score: not measured (no stress test prints a Score: line)
Status: 🟢 PASS
Threshold: 270/300

X Stress test passed

@dmitrii-f-t27
dmitrii-f-t27 force-pushed the ci/brain-unit-cache-key branch from 6c8cc6d to 277e06b Compare October 10, 2026 00:42
@tri-claw

tri-claw Bot commented Oct 10, 2026

Copy link
Copy Markdown

Summary by t27-bees

This PR fixes two CI cache problems: (1) actions/checkout was running after mlugg/setup-zig in all 8 Zig jobs, wiping the restored .zig-cache before builds could use it; (2) the brain-unit matrix job's 5 region legs shared one cache key, causing 4-of-5 saves to be refused and every leg to restore the wrong region's build. The fix reorders checkout before setup-zig everywhere, adds a per-region cache-key to brain-unit, and introduces a gate workflow + checker to prevent regressions.

Worth a look

  • .github/workflows/brain-ci.yml:252–256 — cache-key: ${{ matrix.region }} is added only to brain-unit. If any other job later gains a matrix that compiles region-specific artifacts, it will need the same treatment or will suffer the same collision/refusal problem.

  • tools/check_setup_zig_order.py:58–60 — restores_cache treats any use-cache value other than literal "false" (case-insensitive) as caching, including expressions like ${{ matrix.c }}. An expression that evaluates to false at runtime will still be treated as caching here, causing a false-positive failure (the test case "use-cache from an expression counts as caching" expects rc 1). This is a deliberate conservative choice but worth knowing.

  • tools/check_setup_zig_order.py:83 — WORKSPACE_PATHS matches "", ".", "./", and "${{ github.workspace }}". A checkout with path: ${{ github.workspace }}/ (trailing slash) or path: . (trailing space) would not match and would pass silently, even though it wipes the workspace. The set could be normalized (strip, rstrip /) for robustness.

  • tools/check_setup_zig_order.py:124 — Exit code 2 ("cannot tell") is returned when no job uses mlugg/setup-zig. This makes the gate fail on repos/workflows that don't use setup-zig at all, requiring them to either add the action or adjust the gate. The comment acknowledges this ("a pass would mean nothing"), but it may surprise users adding the gate to a new repo.

  • .github/workflows/zig-cache-order.yml:25–26 — The concurrency group uses github.workflow_ref plus a run-id discriminator for workflow_dispatch. On a workflow_dispatch re-run, github.run_id changes, so a new run won't cancel the previous one (which may still be running). This is likely intentional for manual dispatches, but note that rapid manual re-runs can run concurrently.


Read: 4 files, +308 -40, the whole diff. Head 277e06b. Model: openai-compatible/nvidia/nemotron-3-ultra-550b-a55b. Commands: @t27-bees review | summary | help | pause | resume. Free for every repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant