Skip to content

api | web | docs: delete overwritten build caches in registry cleanup - #51

Merged
bbedward merged 1 commit into
masterfrom
motile/fix-registry-cleanup
Oct 23, 2026
Merged

bbedward merged 1 commit into
masterfrom
motile/fix-registry-cleanup

Conversation

@bbedward

Copy link
Copy Markdown
Contributor

Problem

The registry grows by tens of GB without any new service, and cleanup says there is nothing it can delete.

Every build exports its BuildKit cache (mode=max, every intermediate layer) to one tag per service: <repo>:<config hash>-buildcache. The next build of that service overwrites the tag. The previous cache manifest stays in the repository with no tag pointing at it:

  • cleanup built its inventory from the tag list, so it never saw these manifests
  • registry garbage-collect without --delete-untagged keeps every manifest a repository still holds, so their blobs never went away

With auto-deploys pushing 10 to 20 builds a day, that is a dead cache per build, forever. Growing the volume only buys time.

Fix

Cleanup now also lists the manifests each repository stores on disk (_manifests/revisions). A manifest no tag and no tagged index points at is deleted before any tagged image, whenever cleanup runs past the threshold check. The one hour grace still applies, since a push writes the manifest before the tag. Children of a tagged index are never treated as orphans.

The result reports them as deleted_untagged, and the Registry tab shows "N images and M overwritten caches".

Verification

On a dev registry, with the released unbind:latest cleanup image and then with this branch:

before after freed
released cleanup, --all 145,948 KiB 145,948 KiB 52 KiB
this branch, --all 145,948 KiB 74,188 KiB 70 MiB (the overwritten cache)
this branch, index image with attestations newest 217,952 KiB 74,404 KiB overwritten cache + older image; index kept its children and pulls

Unit tests cover untagged manifests going first and always, the fresh grace, the newest-per-repo rule ignoring them, orphan detection against tagged index children, and parsing the revision listing.

Every build exports its cache to one tag per service, so each build
overwrites the previous cache. The old cache manifest stays in the
repository untagged, cleanup only looked at tags, and garbage collection
keeps every manifest a repository still holds. The volume filled up with
caches nothing could reach and cleanup reported nothing prunable.

Cleanup now lists the manifests each repository stores, treats the ones no
tag or tagged index points at as the first deletions, and keeps the one
hour grace for a manifest whose tag is still being pushed. The result
counts them separately and the Registry tab shows the count.

Measured on a dev registry: two builds left 70 MiB in an overwritten
cache, the old cleanup freed 52 KiB, the new one frees all of it. An index
image with attestations keeps its untagged children and still pulls.
@bbedward
bbedward merged commit cd73bc0 into master Oct 23, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant