ci: setup-zig caches fill the 10 GB Actions cache and evict every chipdb #844

Activity

1. gHashTag

   added

   bugSomething isn't workingSomething isn't working

   P1High priorityHigh priority

   on Oct 4, 2026
2. gHashTag commented on Oct 5, 2026

   

   gHashTag

   on Oct 5, 2026

   OwnerAuthor

   More actions

   Measured: what each Zig job gains from its restored cache

   This answers the "Not measured" line above. Read from job logs with speed zigcache (the build-speed loop's ledger tool), which pairs each restore with the run that saved the entry, by key.

   brain-ci.yml: the hits were never used

   All eight Zig jobs ran mlugg/setup-zig before actions/checkout. setup-zig restores into $GITHUB_WORKSPACE/.zig-cache; checkout then logs Deleting the contents of '<workspace>'. Last 25 runs: 238 restores, 238 hits, 0 used, and every job still saved a new entry.

   Live list at 2026-10-05 01:05Z: 178 entries, 10.83 GB. setup-zig-cache-v2-* is 174 of them (10.55 GB, 97%); brain-ci's jobs alone are 161 (8.43 GB, 78%).

   Fix: #890 (checkout first). One dispatch of it, all green: the eleven Zig jobs' work after setup-zig summed 170.5 s against 477.7 s for the cold median. Not a controlled A/B; numbers and caveats are in the PR.

   ci.yml Build & Test: the cache nets about nothing

   38 jobs: 12 warm, 26 cold. All 26 cold jobs had restored an entry of about 190 B: setup-zig empties a directory over cache-size-limit (2 GiB) and saves it anyway, so the run after a big build starts cold. Of the 12 warm jobs, 4 reused their entry (work 25.7–30.4 s, directory stayed 1.53 GB) and 8 rebuilt everything (57.8–82.6 s, 1.53 → 3.01 GB, then emptied). Net −1.7 s per job against cold.

   Why a warm entry rebuilds: the runner CPU model

   build.zig uses standardTargetOptions, so the target is the native CPU, which is in every artifact's hash. ubuntu-latest jobs in one run land on AMD EPYC 7763, 9V45, 9V74 and Intel Xeon Platinum 8573C.

   Probe branch exp/zig-cpu-probe (#890 plus a step printing /proc/cpuinfo's model name), runs 37249467137 → 37249915131:

   restorer vs saver	result
   same CPU model	Health, Metrics, Build Check: 3 of 3 reused, 0.6–2.9 s, directory unchanged
   other CPU model	CLI Smoke, Integration, Unit locus-coeruleus grew (0.39 → 0.70, 0.23 → 0.32, 0.17 → 0.23 GB). Stress (9V45 → 7763) did not grow, which is not explained

   Negative control: the runner image version was the same (20260927.320.1) on every pair, and on all 9 saver/restorer pairs in 40 ci.yml runs, 7 of which rebuilt.

   Pinning the CPU: measured once

   Branch exp/zig-dcpu (the probe plus -Dcpu=x86_64_v3 on all nine zig build lines in brain-ci.yml). All four runner CPUs support x86_64_v3. Gen 1 run 37251000354 saved, gen 2 run 37251511253 restored, both green:

   Saver and restorer are matched by key (setup-zig-cache-v2-<job id>-…-37251000354-1), so each row is the same job in both runs.

   job (saver → restorer CPU)	command	work after setup-zig	.zig-cache GB	native target, probe
   Health (7763 → 7763, same)	tri-compile	2.7 s	0.70 → 0.70	same CPU: unchanged
   Integration (9V74 → 7763)	test-brain	0.7 s	0.23 → 0.23	7.6 s, 0.23 → 0.32
   Stress (9V74 → 7763)	test-brain-stress	31.7 s, mostly running	0.23 → 0.23	did not grow either
   CLI Smoke (9V74 → 9V45)	tri-compile	8.1 s	0.70 → 0.74	25.6 s, 0.39 → 0.70
   Build Check (9V74 → 9V45)	tri-compile	4.9 s	0.73 → 0.78	same CPU in the probe
   Metrics (Intel 8573C → 9V74)	tri-compile	8.3 s	0.69 → 0.74	same CPU in the probe

   With the CPU pinned, an entry saved on one model was reused on another. The two test jobs reused all of it. The three other-CPU tri-compile jobs each added about 45 MB (0.045 GB) and took 4.9–8.3 s, against 23–27 s for the same jobs when gen 1 rebuilt them from native-target entries. What those 45 MB are is not explained. They are not the build runner: the two test jobs run zig build too and did not grow. The five Unit legs are not counted, because they still shared one key and restored one region's entry.

   Caveats: one pair of runs, with different CPU pairs than the probe; the OS and glibc part of the triple stays host-detected; and a pinned CPU changes what vsa_simd is compiled for, so whether to pin is the owner's call.

   Matrix legs share one key

   brain-unit's five legs get the same key, so one saves and the rest are refused: 88 of 110 saves in 25 runs. In the probe, all five legs restored the locus-coeruleus leg's entry. cache-key: ${{ matrix.region }} separates them; branch ci/brain-unit-cache-key, stacked on #890, is being measured.

   Not measured

   • Stress's lack of growth across CPU models with the native target.
   • What the ~45 MB is that each other-CPU tri-compile job still adds with -Dcpu pinned.
   • Whether pinning slows vsa_simd or any other code that tuned itself to the host CPU.
   • How many runs a brain-ci job takes to reach the 2 GiB limit once its hits are used.
   • Anything about chipdb hit rate after ci(cache): prune Actions cache entries nothing can restore #853's prune; fix(ci): key the chipdb cache on the openXC7 image so it can hit #852 is the chipdb fix.

   🤖 Generated with Claude Code
3. gHashTag

   added a commit that references this issue on Oct 5, 2026

   ci(cache): test the prune rules on the pull request that changes them

   ...

   5ea1f1c
4. gHashTag commented on Oct 5, 2026

   

   gHashTag

   on Oct 5, 2026

   Last edited by gHashTag

   OwnerAuthor

   More actions

   Measured on 2026-10-05: #853's prune is merged and working, but it was not enough to keep a chipdb on a branch.

   • At its 01:13Z run the prune kept 130 entries, 7.97 GB. By 03:54Z the cache was at 165 entries, 11.11 GB. Of a 162-entry list taken after 03:45Z, setup-zig held 96.7% (10.48 of 10.84 GB): brain-ci's seven Zig jobs 8.59 GB, CI's test 1.85 GB.
   • The hourly schedule (41 * * * *) started 2 runs in the 9 slots between the 18:41Z merge and 03:46Z.
   • Past 10 GB GitHub evicts by last read. The oldest last read left in the cache was 22:42Z, about 5 h old. The chipdb saved on fix/ci-chipdb-cache on 2026-10-04 was gone. The LED one-hot run on fix(ci): key the chipdb cache on the openXC7 image so it can hit #852 (37260180318) found no cache and rebuilt it in 2 m 54 s.

   #895 adds an 8 GB budget, spending only setup-zig's per-run entries off main, least recently read first. It also prunes after brain-ci and CI complete. Dry run on the list above: keep 119 entries (7.78 GB), including all 5 chipdb entries and all 9 on main. Not measured: the workflow_run trigger cannot fire before merge.

   🤖 Generated with Claude Code
5. dmitrii-f-t27

   added a commit that references this issue on Oct 10, 2026

   ci(cache): prune to an 8 GB budget, and after brain-ci and CI

   ...

   6ab73d0