Repository navigation
Caching is inconsistent in FFI CI #1131
Description
We pin the rust version so the cache should always hit as long as the rust dependencies haven't changed.
Here is an example where cache was hit: https://github.com/payjoin/rust-payjoin/actions/runs/18139263858/job/51625878478
And two where it wasn't (despite the cache keys matching exactly, I would expect at least one of these to hit):
Activity
emmanuelist commentedon Aug 15, 2026
Had a dig at this one. The keys are fine. The cache is just getting evicted
before anything can hit it.
Where the repo cache sits right now:
GitHub is evicting least-recently-used entries continuously.
refs/pull/1809/merge.A single PR's run is holding the whole quota.
refs/heads/masterat all. Querying?ref=refs/heads/mastergives back
total_count: 0.What's actually taking up the space:
.nar.z).narinfo-*v0-rust-*(the build caches)magic-nix-cache-actionwrites one entry per nix store path, so roughly 1016nix entries are crowding out the 15
Swatinem/rust-cacheentries that the jobsactually want back.
I think that accounts for the inconsistency. Cache scope is per-ref: a PR can
restore from its own ref or from the default branch, never from another PR.
Master has nothing on it, so every PR starts cold, and whether a re-run hits
comes down to whether someone else's nix flood evicted your entry in the
meantime. Same key, different outcome, depending who ran last.
It doesn't help that
dart.yml,javascript.ymlandpython.ymlonly fire onpull_request, andcsharp.ymlotherwise only on tag pushes. None of the fourever runs on a push to master, so none of them can seed a cache for PRs to
inherit.
rust.ymlandformat.ymldo run on push, and even those have seedednothing, which I think just confirms the eviction.
@benalleng you flagged this in #1798 already ("Perhaps it is worth exploring
whether or not this job is totally worth it if our repo immediately fills the
cache space"). It does fill it, more or less immediately.
Options as I see them:
magic-nix-cache-actionfrom the FFI workflows and lean onSwatinem/rust-cache. Smallest change, frees the quota straight away, butwe give up nix caching.
nix-community/cache-nix-action,which keeps the store as one bounded entry instead of thousands.
I don't have a strong preference, and option 1 does trade away something real,
so I'd rather you pick. Happy to write up whichever one you want.
Disclosure: I used Claude Code to pull these numbers and draft this comment.
emmanuelist commentedon Aug 18, 2026
Correction on my numbers above. I measured entry counts and then wrote as if I had
measured bytes, and it turns out that matters a lot here.
Actual split right now, fully paginated and cross-checked against
/actions/cache/usage:.nar.z/.narinfo)v0-rust-*)So the nix entries are around 0.7 MB each and the rust-cache ones around 635 MB. All
those thousands of nix entries are noisy but they are not what fills the quota.
rust-cache is doing that pretty much on its own.
The parts I had right still stand: magic-nix-cache does write one entry per store
path, cache scope is per-ref, and master has nothing on it for PRs to inherit. What I
got wrong was saying the nix entries were crowding rust-cache out. They weren't.
Which means #1814 takes the entry count right down and stops the per-store-path
churn, but it won't stop the eviction this issue is actually about. I should have
checked that before opening it.
The lever that does look right is
save-ifon rust-cache:None of the 13 rust-cache call sites set it, so every PR run saves its own copy, and
those are running 0.1 to 1.5 GiB each. Something like:
would have PRs restore and only master save. That drops the PR-scoped entries, and
as a side effect it finally puts a cache on master for PRs to inherit, which is the
other half of what's broken here.
Downside is a PR that changes dependencies gets a colder build until master catches
up, so it's a real tradeoff rather than a free win. Happy to open it as a separate PR
once #1814 is in, if you want to go that way.
Disclosure: I used Claude Code to pull these numbers and draft this comment.
benalleng commentedon Aug 18, 2026
I think for any PR that changes the dep tree it is ok to not get the full benefit of caching.
Bound nix cache size in CI
Bound nix cache size in CI
cderinbogaz commentedon Sep 7, 2026
Disclosure: I work on Runnerhut. The corrected byte totals matter more than the original entry-count diagnosis. For #1824, I would verify a successful default-branch writer actually runs for each FFI cache family; making PRs restore-only cannot create a default-branch cache if that workflow never executes there. Then test an unchanged-dependency PR and a dependency-bump PR, tracking bytes, hit rate and build time. This checks the accepted cold-dependency tradeoff without claiming a new runner will change GitHub's cache scoping.