Skip to content

Build the CI MinIO container from the GHCR mirror (PP-5244) - #3767

Merged
dbernstein merged 1 commit into
mainfrom
chore/minio-mirror-switch
Sep 25, 2026
Merged

dbernstein merged 1 commit into
mainfrom
chore/minio-mirror-switch

Conversation

@dbernstein

@dbernstein dbernstein commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Description

Switches docker/Dockerfile.minio.ci from quay.io/minio/minio:latest, which no longer permits
anonymous pulls, to the Palace mirror:

-FROM quay.io/minio/minio:latest AS minio
+# MinIO withdrew anonymous public access to its own image from both Docker Hub and quay.io, so
+# this pulls from the Palace mirror of the upstream release instead. The mirror is an unmodified
+# MinIO build, published from the ci-scripts repo:
+# https://github.com/ThePalaceProject/ci-scripts/blob/main/images/minio/Dockerfile
+# The tag is pinned deliberately; the mirror publishes no `latest`.
+FROM ghcr.io/thepalaceproject/palace-ci-minio:RELEASE.2025-09-07T16-13-09Z AS minio

That is the whole change: one FROM line, plus a comment recording why it points somewhere
unusual. The mirror is an unmodified MinIO build, so the credentials, ports and command below the
FROM are untouched.

The mirror is published and public: ghcr.io/thepalaceproject/palace-ci-minio is live at digest
sha256:0e8998d7ba6f742bac3c6ad22f354979b987d5d31c0a95b9b0dcad9d73e32751 with linux/amd64 and
linux/arm64, and an anonymous (unauthenticated) manifest request returns 200.

It is published from ci-scripts rather than from
here, because the image is shared by three repos and belongs to none of them in particular.
Companion one-line changes: ThePalaceProject/library-registry#1066 and
ThePalaceProject/virtual-library-card#1016. The three are independent and can merge in any order.

Motivation and Context

JIRA

All tox-docker CI jobs currently fail on main while building the MinIO test container:

docker.errors.BuildError: unauthorized: access to the requested resource is not authorized

MinIO has withdrawn anonymous public access to its server image from Docker Hub (~13 Sept, worked
around in #3728 by moving to quay.io) and now from quay.io as well.
Ten checks fail on main: Tests (Py 3.12 / 3.13 / 3.14), OpenSearch 2.19, OpenSearch 3.5, Unit
tests (amd64 / arm64), Integration test (amd64 / arm64) and the Backwards compatibility test. Lint,
mypy, CodeQL, the Migration test and the Docker build are unaffected because they do not use
tox-docker. Local tox fails identically, so developers cannot run the suites either.

Pinning an older tag does not help — the whole upstream repository is gated, not just :latest —
and neither does a local Docker cache, because tox_docker passes pull=True on every build.

Full background, registry probes and the retirement plan are in ThePalaceProject/ci-scripts#2.

How Has This Been Tested?

This PR's own CI is the acceptance test, and it passes. All ten checks listed above are green
against the published mirror, including the Backwards compatibility test and both Integration and
Unit test jobs on amd64 and arm64.

Locally, tox -e py312-docker -- -m minio was run against the real
ghcr.io/thepalaceproject/palace-ci-minio reference with the image first cleared from the local
Docker cache, so the pull genuinely came from GHCR:

py312-docker: docker> build .../docker/Dockerfile.minio.ci target 'minio'
py312-docker: docker> run '2827512c875a' (from 'minio-circ')
py312-docker: commands[0]> pytest -m minio --no-cov -q
14 passed in 16.48s

Before the mirror was published, the same chain was exercised by pushing the image to a throwaway
registry:2 on localhost and temporarily pointing this FROM line at it — tox_docker passes
pull=True on every build, so a local-only tag is rejected and a real registry is required either
way. That run also gave 14 passed.

Checklist

  • I have updated the documentation accordingly.
  • All new and existing tests passed.

🤖 Generated with Claude Code

@dbernstein dbernstein mentioned this pull request Sep 24, 2026
2 tasks done
@claude

claude Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @dbernstein's task in 56s —— View job


Summary

I found nothing to change. The PR swaps the unpullable quay.io/minio/minio:latest base for the pinned GHCR mirror (ghcr.io/thepalaceproject/palace-ci-minio:RELEASE.2025-09-07T16-13-09Z) and adds a comment explaining why the tag is pinned. docker/Dockerfile.minio.ci is used in two places. One is tox's [docker:minio-circ] in tox.ini. The other is the minio service in docker-compose.yml, which docker/ci/test_backwards_compatibility.sh uses. Both pick up the new image with no other edits. Nothing else in the repo refers to quay.io or minio/minio. Dependabot doesn't watch the Docker ecosystem, so it won't try to bump the pinned tag. The Greptile "not safe to merge" finding is about src/palace/manager/sqlalchemy/model/work.py (classification repair), and it also mentions SAML, a migration and dependency changes. None of those are in this diff: git diff origin/main...HEAD touches only docker/Dockerfile.minio.ci. Greptile seems to have reviewed a different diff, so its finding doesn't apply to this PR. The tox-docker Test workflow for this commit was still running when I checked, so CI hasn't yet confirmed the fix.

@dbernstein
dbernstein force-pushed the chore/minio-mirror-switch branch from e3bdc75 to dff8b78 Compare September 24, 2026 18:26
@greptile-apps

greptile-apps Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

The PR appears safe to merge.

Summary

The PR switches the CI MinIO image from quay.io to the Palace GHCR mirror at a pinned release tag. There have been no changes since the previous review.

Reviews (3) · Last reviewed commit: "Build the CI MinIO container from the GH..."

MinIO has withdrawn anonymous public access to its server image from both Docker
Hub and quay.io, so `FROM quay.io/minio/minio:latest` now fails with a 401 before
the test suite starts. Pinning an older tag does not help (the whole repository
is gated), and neither does a local cache (tox_docker passes pull=True on every
build).

Points the CI MinIO container at the Palace mirror instead, published from the
ci-scripts repo. The mirror is an unmodified MinIO build, so the credentials,
ports and command below the FROM line are untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@dbernstein
dbernstein force-pushed the chore/minio-mirror-switch branch from dff8b78 to 374f27c Compare September 24, 2026 19:16
@dbernstein
dbernstein changed the base branch from chore/minio-mirror-image to main September 24, 2026 19:16
@greptile-apps

greptile-apps Bot commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Comments Outside Diff

These findings sit on lines the diff does not cover, so they could not be posted inline. Each one leaves this list once its file changes.

  • P1 Incorrect nonfiction status persists src/palace/manager/sqlalchemy/model/work.py:1413 ▶

    When the repair reclassifies a BISAC subject that was wrongly marked nonfiction, that subject may now provide no fiction-status evidence. If it was the Work’s only such evidence, these lines pass the Work’s old False value back as the classifier default. The subject stops voting nonfiction, but the Work remains incorrectly marked nonfiction, and its genres are filtered using that incorrect status.

dbernstein added a commit to ThePalaceProject/ci-scripts that referenced this pull request Sep 24, 2026
## Description

Adds a mirror of the upstream MinIO server image, published as
`ghcr.io/thepalaceproject/palace-ci-minio`:

- `images/minio/Dockerfile` — assembles the image from MinIO's official
GitHub release binaries
(`minio` `RELEASE.2025-09-07T16-13-09Z`, `mc`
`RELEASE.2025-08-13T08-35-41Z`) plus the AGPL
  license files, each pinned by sha256 via BuildKit's `ADD --checksum`.
- `.github/workflows/build-minio-mirror.yml` — builds `linux/amd64` +
`linux/arm64` and pushes to
GHCR. Runs only when the Dockerfile or the workflow changes, or on
demand.
- A README section covering what the image is and how to consume it.

This lives in ci-scripts rather than circulation because the image is
shared by three repos and
belongs to none of them in particular. GHCR packages are namespaced by
org, not repo, so one mirror
serves all three. Supersedes ThePalaceProject/circulation#3766, which is
now closed.

### Design notes

- **Assembled from release binaries, not pulled and re-tagged.** Both
upstream registries are
closed to anonymous pulls, so there is no image left to `docker pull`,
and `dl.min.io` is gone
too (410). GitHub release assets are the one channel MinIO still serves
anonymously. The binaries
are byte-for-byte identical to the ones inside the last upstream image
CI used — verified below.
- **Bare passthrough, no Palace configuration.** circulation and
library-registry use one set of credentials, the library registry uses
another. Baking any of that in would fork the image between repos
immediately, so each repo keeps its own Dockerfile and changes only its
`FROM` line.
- **Pinned to an immutable tag, no `latest`.** Following a moving
upstream tag is part of how we
got here. `RELEASE.2025-09-07T16-13-09Z` is also the last MinIO release
to publish binaries at
all — later tags ship no assets — so there is no newer version to move
to.
- **Only `main` publishes.** Pull requests and `workflow_dispatch` runs
from a branch build both
architectures and verify the checksums, then discard the result. An
unmerged branch must not be
  able to overwrite a tag three other repos pin their CI to.
- **Public visibility**, consistent with `circ-webapp`, `circ-scripts`,
`circ-exec` and
`circ-baseimage`, so no `docker login` for developers and no
package-access grants for CI.
- **`palace-` rather than `circ-` prefix**, because the image is shared
across three repos.

### ⚠️ One-time manual step after merge

GHCR packages are created **private**. Once the workflow has run, set
`palace-ci-minio` to public
(Org → Packages → `palace-ci-minio` → Package settings). Until that is
done, the three consumer PRs
will still fail.

Consumer PRs, all draft, each a single `FROM` line:
ThePalaceProject/circulation#3767,
ThePalaceProject/library-registry#1066,
ThePalaceProject/virtual-library-card#1016.

## Motivation and Context

All tox-docker CI jobs in circulation fail while building the MinIO test
container, and the other
two repos will hit the same wall on their next run:

```
docker.errors.BuildError: unauthorized: access to the requested resource is not authorized
```

MinIO has progressively withdrawn public distribution — Docker Hub
around 13 Sept (worked around in
ThePalaceProject/circulation#3728 by moving to quay.io), and quay.io
around 24 Sept. Anonymous
probes on 24 Sept:

| Probe | Result |
| --- | --- |
| `quay.io/minio/minio:latest` manifest | 401 |
| `quay.io/minio/minio:RELEASE.2024-01-16T16-07-38Z` (pinned old tag) |
401 |
| `registry-1.docker.io/minio/minio:latest` manifest | 401 |
| Docker Hub API for `minio/minio` | `object not found` |
| `dl.min.io` server/client binaries | 410 |
| `quay.io/prometheus/busybox:latest` (control) | 200 |

Quay issues an anonymous pull token and then refuses the manifest, so
the repository is gated rather
than the network; the control confirms anonymous pulls work from the
same machine. Two consequences:
pinning a digest or an older tag will not help, because the whole
repository is closed; and a local
cache will not help, because `tox_docker` passes `pull=True` on every
build
(`tox_docker/plugin.py`), forcing a fresh pull. Local `tox` fails
identically, so developers cannot
run the suites either.

### Retirement

This mirror is a bridge, not a destination — the intent is to drop MinIO
for a maintained
S3-compatible image. It should be short-lived for two reasons: we do not
want to become a de-facto
public distributor of a frozen MinIO build; and GitHub does not allow
self-service deletion of a
public package once any version exceeds 5,000 downloads, above which it
becomes a Support request.
With `pull=True` on every build, ephemeral runners and three repos
pulling, that threshold arrives
quickly.

When the time comes:

```
gh api -X DELETE /orgs/ThePalaceProject/packages/container/palace-ci-minio
```

## How Has This Been Tested?

Verified locally on macOS / Docker 29.6.1 (arm64 host).

**The mirrored artifacts are identical to the withdrawn upstream
image.** A cached copy of
`quay.io/minio/minio:latest` (labelled `RELEASE.2025-09-07T16-13-09Z`)
was still present locally;
its binaries and license files hash the same as what the mirror
downloads:

```
# inside the cached upstream image (arm64)          # published by MinIO on GitHub
5c83cd2c…f03d  /usr/bin/minio                        5c83cd2c…f03d  minio.linux-arm64.RELEASE.2025-09-07T16-13-09Z
14c8c961…c12c  /usr/bin/mc                           14c8c961…c12c  mc.linux-arm64.RELEASE.2025-08-13T08-35-41Z
0d96a4ff…bcb0  /licenses/LICENSE                     0d96a4ff…bcb0  minio/minio@RELEASE.2025-09-07T16-13-09Z:LICENSE
113b8c63…8542  /licenses/CREDITS                     113b8c63…8542  minio/minio@RELEASE.2025-09-07T16-13-09Z:CREDITS
```

**Build.** Builds for `linux/amd64` and `linux/arm64`; both
architectures' checksums verify (a
mismatch fails the build by design). The final stage has no `RUN`
instructions, so it cross-builds
without QEMU — confirmed on a CI runner that advertised only
`linux/amd64…/386` as supported.

**Runtime.** `minio --version` and `mc --version` report the expected
releases; the server comes up
and `/minio/health/live` returns 200 in ~3s; the console on :9001
returns 200; `curl` inside the
container returns 200 against the health endpoint (the check
`docker-compose.yml` uses); and
`mc alias set` / `mc mb` / `mc anonymous set download` all succeed (the
operations
virtual-library-card's entrypoint needs). `/licenses/LICENSE` and
`/licenses/CREDITS` are present
and hash-identical to upstream's.

**End-to-end through tox.** Because `pull=True` rejects a local-only
tag, the image was pushed to a
throwaway `registry:2` on localhost and circulation's
`docker/Dockerfile.minio.ci` temporarily
pointed at it, so the whole chain was exercised — tox building the
consumer Dockerfile, pulling the
mirror from a real registry, starting the container and running the
tests:

```
py312-docker: docker> build .../docker/Dockerfile.minio.ci target 'minio'
py312-docker: docker> run 'sha256:fe9e8121edcb' (from 'minio-circ')
py312-docker: commands[0]> pytest -m minio --no-cov -q
14 passed in 13.22s
```

The same local-registry substitution was used to build and run
library-registry's
`docker/Dockerfile.minio.ci` (healthy, `mc` works) and
virtual-library-card's `ci/minio/Dockerfile`
(entrypoint completes — bucket created, anonymous download policy set,
anonymous bucket read returns
200).

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@dbernstein
dbernstein marked this pull request as ready for review September 24, 2026 20:12
@dbernstein dbernstein changed the title Build the CI MinIO container from the GHCR mirror Build the CI MinIO container from the GHCR mirror (PP-5244) Sep 24, 2026
@dbernstein dbernstein closed this Sep 24, 2026
@dbernstein dbernstein reopened this Sep 24, 2026
@dbernstein
dbernstein requested a review from a team September 24, 2026 20:21
@codecov

codecov Bot commented Sep 24, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.71%. Comparing base (baa5125) to head (374f27c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3767      +/-   ##
==========================================
- Coverage   93.72%   93.71%   -0.01%     
==========================================
  Files         510      510              
  Lines       46510    46510              
  Branches     6313     6313              
==========================================
- Hits        43590    43589       -1     
- Misses       1886     1887       +1     
  Partials     1034     1034              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@dbernstein

Copy link
Copy Markdown
Contributor Author

Comments Outside Diff

These findings sit on lines the diff does not cover, so they could not be posted inline. Each one leaves this list once its file changes.

  • P1 Incorrect nonfiction status persists src/palace/manager/sqlalchemy/model/work.py:1413 ▶
    When the repair reclassifies a BISAC subject that was wrongly marked nonfiction, that subject may now provide no fiction-status evidence. If it was the Work’s only such evidence, these lines pass the Work’s old False value back as the classifier default. The subject stops voting nonfiction, but the Work remains incorrectly marked nonfiction, and its genres are filtered using that incorrect status.

This shouldn't be here. I'm talking to Claude about how it might have popped up. Regardless it has nothing to do with this PR.

@dbernstein
dbernstein merged commit a87d4c7 into main Sep 25, 2026
32 of 49 checks passed
@dbernstein
dbernstein deleted the chore/minio-mirror-switch branch September 25, 2026 02:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants