Repository navigation
api | web | docs: delete overwritten build caches in registry cleanup - #51
Merged
Merged
Conversation
Every build exports its cache to one tag per service, so each build overwrites the previous cache. The old cache manifest stays in the repository untagged, cleanup only looked at tags, and garbage collection keeps every manifest a repository still holds. The volume filled up with caches nothing could reach and cleanup reported nothing prunable. Cleanup now lists the manifests each repository stores, treats the ones no tag or tagged index points at as the first deletions, and keeps the one hour grace for a manifest whose tag is still being pushed. The result counts them separately and the Registry tab shows the count. Measured on a dev registry: two builds left 70 MiB in an overwritten cache, the old cleanup freed 52 KiB, the new one frees all of it. An index image with attestations keeps its untagged children and still pulls.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The registry grows by tens of GB without any new service, and cleanup says there is nothing it can delete.
Every build exports its BuildKit cache (
mode=max, every intermediate layer) to one tag per service:<repo>:<config hash>-buildcache. The next build of that service overwrites the tag. The previous cache manifest stays in the repository with no tag pointing at it:registry garbage-collectwithout--delete-untaggedkeeps every manifest a repository still holds, so their blobs never went awayWith auto-deploys pushing 10 to 20 builds a day, that is a dead cache per build, forever. Growing the volume only buys time.
Fix
Cleanup now also lists the manifests each repository stores on disk (
_manifests/revisions). A manifest no tag and no tagged index points at is deleted before any tagged image, whenever cleanup runs past the threshold check. The one hour grace still applies, since a push writes the manifest before the tag. Children of a tagged index are never treated as orphans.The result reports them as
deleted_untagged, and the Registry tab shows "N images and M overwritten caches".Verification
On a dev registry, with the released
unbind:latestcleanup image and then with this branch:--all--allUnit tests cover untagged manifests going first and always, the fresh grace, the newest-per-repo rule ignoring them, orphan detection against tagged index children, and parsing the revision listing.