Drop Python 3.11 support and promote clodius to a core dependency

.github/workflows/test.yml

@@ -19,7 +19,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.11", "3.12", "3.13"]
+        python-version: ["3.12", "3.13"]
     steps:
       - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1
 

Makefile

@@ -52,62 +52,6 @@ api:
 	docker run -d --name api --network cvh-backend-network --network-alias cvh-backend -p 8000:8000 -e SYNC_DATA_DIR=/tmp/sync-data api
 	@echo "API container is up and running on port 8000 (http://0.0.0.0:8000/metadata)."
 
-# --- Local-dev matrix tile serving (issue #82) -----------------------
-#
-# `make api` builds an image with no clodius, so its tile routes answer
-# 501 -- correct for production while the `tiles` extra stays optional,
-# but it leaves the whole tile chain untestable through the documented
-# Docker flow. These targets layer clodius on (installed from the public
-# fork at the sha pyproject.toml pins) and add the piece `make api` does
-# not need: a cache volume shared with a worker, since a tileset
-# artifact is built by the worker and then opened by the API.
-#
-# Retire both once `tiles` becomes an ordinary dependency (raising the
-# Python floor to 3.12) -- `make api` then covers this on its own.
-
-api-tiles:
-	make network
-	@docker stop api 2>/dev/null || true
-	@docker rm api 2>/dev/null || true
-	@echo "Building the API image..."
-	docker build -t api -f Dockerfile.api .
-	@echo "Layering the clodius tile backend on top..."
-	docker build -t api-tiles -f Dockerfile.api-tiles .
-	@# The image runs as `app`, but a freshly created named volume is
-	@# owned by root, so the lifespan's mkdir under SYNC_DATA_DIR fails
-	@# with PermissionError and the API 500s on every workflow route.
-	@docker volume create cfdb-sync >/dev/null
-	@docker run --rm --user 0 --entrypoint chown -v cfdb-sync:/tmp/sync-data api-tiles -R app:app /tmp/sync-data
-	@echo "Starting the API container with matrix tile serving enabled..."
-	docker run -d --name api --network cvh-backend-network --network-alias cvh-backend -p 8000:8000 \
-		-e SYNC_DATA_DIR=/tmp/sync-data \
-		-e WORKFLOW_POOL_NAMESPACE=$${WORKFLOW_POOL_NAMESPACE:-cfdb-workers} \
-		-v cfdb-sync:/tmp/sync-data \
-		api-tiles
-	@echo "API up on port 8000."
-	@# Start the worker FIRST next time: the API dispatches on request, and a
-	@# dispatch that finds no worker is queued rather than failed, so the job
-	@# then waits out CFDB_WORKFLOW_RETRY_INTERVAL_S (2 min) before running.
-	@echo "Run 'make worker-tiles' before POSTing a tileset, or the first job waits ~2 min for the retry tick."
-
-# The worker that builds tileset artifacts. It reuses the API image
-# rather than `cfdb-wool`, because the matrix processor needs cooler and
-# h5py, which arrive with clodius and are absent from the worker image.
-# It shares the API's cache volume: the worker writes the artifact, the
-# API opens it locally with h5py.
-worker-tiles:
-	make network
-	@docker stop worker 2>/dev/null || true
-	@docker rm worker 2>/dev/null || true
-	@echo "Starting a containerized LAN worker pool (namespace=$${WORKFLOW_POOL_NAMESPACE:-cfdb-workers})..."
-	docker run -d --name worker --network cvh-backend-network \
-		-e SYNC_DATA_DIR=/tmp/sync-data \
-		-e WORKFLOW_POOL_NAMESPACE=$${WORKFLOW_POOL_NAMESPACE:-cfdb-workers} \
-		-e WORKFLOW_WORKER_COUNT=$${WORKFLOW_WORKER_COUNT:-1} \
-		-v cfdb-sync:/tmp/sync-data \
-		api-tiles python -m cfdb.workflows.worker_lan
-	@echo "Worker container is up. Check logs with: docker logs -f worker"
-
 schema:
 	@echo "Regenerating schema.graphql from the Strawberry schema..."
 	uv run python scripts/export_schema.py

README.md

@@ -10,15 +10,9 @@ CFDB is a Python package for querying and serving enriched C2M2 (Crosscut Metada
 pip install git+https://github.com/abdenlab/cfdb.git
 ```
 
-Requires Python 3.11 or later.
+Requires Python 3.12 or later.
 
-Matrix tile serving for Hi-C contact maps is an optional extra, because the [`abdenlab/clodius`](https://github.com/abdenlab/clodius) fork it depends on requires Python 3.12 while cfdb still supports 3.11. The dependency is pinned to a commit sha of the public fork, so enabling it needs nothing beyond the extra:
-
-```bash
-uv sync --extra tiles
-```
-
-Without it the API runs unchanged and the tile endpoints answer `501`. See [Matrix tile serving](#matrix-tile-serving-hi-c-contact-maps).
+Matrix tile serving for Hi-C contact maps (see [Matrix tile serving](#matrix-tile-serving-hi-c-contact-maps)) is backed by the [`abdenlab/clodius`](https://github.com/abdenlab/clodius) fork, pinned to a commit sha of the public fork as an ordinary dependency — no separate install step is needed beyond the usual `pip install` / `uv sync` above.
 
 ### Environment Variables
 
@@ -930,7 +924,7 @@ Both endpoints accept repeatable `d` parameters, capped at 64 per request. The r
 | 400 | Malformed uid, no `d` parameter, more than 64 `d` parameters, or an unknown DCC |
 | 403 | File requires consortium/protected access (HuBMAP) |
 | 404 | File not found, not a contact map, or its tileset artifact has not been built yet |
-| 501 | This build carries no tile backend (see the `tiles` extra below), or the file is a `.hic` |
+| 501 | This build carries no tile backend (a broken or incomplete install), or the file is a `.hic` |
 | 503 | Workflow subsystem disabled, or the artifact could not be pulled onto local disk in time (`Retry-After`) |
 
 #### Preparing a contact map
@@ -958,16 +952,16 @@ curl http://localhost:8000/tilesets/4dn/<local_id>/status      # {"ready": true}
 
 #### Serving tiles locally
 
-`make api` builds from `Dockerfile.api`, which carries no clodius, so its tile routes answer `501` — correct while the `tiles` extra stays optional, but it leaves the whole chain untestable through the documented Docker flow. `make api-tiles` layers the tile backend on (installed from the public fork at the same sha `pyproject.toml` pins) and adds the one piece `make api` does not need: a cache volume shared with a worker, because a tileset artifact is built by the worker and then opened locally by the API with h5py. Both targets are interim scaffolding and are retired once `tiles` becomes an ordinary dependency, at which point `make api` covers this on its own.
-
-The worker reuses the API image rather than `cfdb-wool`: the matrix processor needs `cooler` and `h5py`, which arrive with clodius and are absent from the worker image.
+`clodius` is an ordinary dependency (see [Installation](#installation)), so `make api` and `make wool` both carry the tile backend directly — there is no separate tile-enabled image to build or retire. Start MongoDB, a worker, and the API the same way any other workflow-subsystem feature is tested locally:
 
 ```bash
 make mongodb        # database
-make worker-tiles   # worker FIRST — see below
-make api-tiles      # API with the tile backend
+make worker-local    # worker FIRST — see below
+uv run uvicorn cfdb.api.main:app --reload  # API, with SYNC_DATA_DIR set to the same path the worker uses
 ```
 
+`make api` remains available for a quick container-only smoke test, but a container's `SYNC_DATA_DIR` is private to that container's filesystem — running the API and a worker in separate containers needs a shared bind mount for the cache to work, same as any other `SYNC_DATA_DIR`-dependent flow (see [Running a local worker pool](#running-a-local-worker-pool) below). Running both as bare `uv run` processes against the same local `SYNC_DATA_DIR` sidesteps that entirely.
+
 Start the worker **before** POSTing a tileset. A dispatch that finds no worker is queued rather than failed, so the job then waits out `CFDB_WORKFLOW_RETRY_INTERVAL_S` (2 min) before the retry tick picks it up — correct behaviour, but a confusing two minutes of `pending` if you were not expecting it.
 
 The database starts empty, so seed a contact-map document before requesting anything: `/tileset_info` and `/tiles` resolve a file by `{submission, local_id}` and need `filename` (the `.mcool` suffix is what routes it), `md5` (the cache key is content-addressed on it), `access_url`, `file_format.name: "HDF5"`, `dcc.dcc_abbreviation`, and `data_access_level: "public"`, plus `genome_assembly` if you want `coordSystem` in the payload. Note that 4DN's `@@download` URLs answer `403` without credentials; use the open-data S3 mirror (`4dn-open-data-public.s3.amazonaws.com`), which the URL allowlist already covers.
@@ -997,8 +991,7 @@ This is invisible to the client, for two reasons that were verified rather than
 
 #### Limitations
 
-- **The clodius tile backend is an optional extra, so deployed images do not carry it.** Tile reading is backed by the [`abdenlab/clodius`](https://github.com/abdenlab/clodius) fork's `tiles_v2` subpackage, installed via `uv sync --extra tiles` from a commit-pinned git reference on the fork's `main` (a sha, deliberately, rather than a floating branch reference — the fork treats `tiles_v2` as unstable and cfdb is its first consumer, so what runs must be exactly what was validated). The extra stays optional because clodius requires Python 3.12 while cfdb still supports 3.11, so `Dockerfile.api` and `Dockerfile.wool` do not install it and images built today answer `501` on the tile routes. Promoting the dependency to an unconditional one — which raises the floor to 3.12 and lets the images carry the backend — is tracked as follow-up work.
-- **ENCODE `.hic` contact maps are not served.** The pinned clodius does not carry a `.hic` tileset at all, which makes this an unconditional `501` today. It would stay unconditional even if a future clodius did carry one: `.hic` support (tracked separately, on a different path than the tiles_v2 backend above) is built on `hictkpy`, whose entry points take `str | os.PathLike` and nothing else. There is no URL, HTTP, or S3 surface in the library, so a `.hic` can only be read from a local copy. Making that copy is not viable at this corpus's scale: 3,903 files totalling ~78 TB, a median of 10.3 GB, and a largest single file of 315 GB — past the 200 GiB ceiling on Fargate ephemeral storage, so it could not be cached on an API task at any budget. Serving them needs either a remote-capable reader (hictk tracks S3 support in [paulsengroup/hictk#395](https://github.com/paulsengroup/hictk/issues/395), currently scoped to Cooler rather than `.hic`) or an explicit size cap accepting partial coverage — an 8 GiB cap would reach 43% of the files, 20 GiB 67%, 50 GiB 91%. That is deferred rather than decided. 4DN `.mcool` files are unaffected.
+- **ENCODE `.hic` contact maps are not served.** The pinned clodius does not carry a `.hic` tileset at all, which makes this an unconditional `501` today. It would stay unconditional even if a future clodius did carry one: `.hic` support is built on `hictkpy`, whose entry points take `str | os.PathLike` and nothing else. There is no URL, HTTP, or S3 surface in the library, so a `.hic` can only be read from a local copy. Making that copy is not viable at this corpus's scale: 3,903 files totalling ~78 TB, a median of 10.3 GB, and a largest single file of 315 GB — past the 200 GiB ceiling on Fargate ephemeral storage, so it could not be cached on an API task at any budget. Serving them needs either a remote-capable reader (hictk tracks S3 support in [paulsengroup/hictk#395](https://github.com/paulsengroup/hictk/issues/395), currently scoped to Cooler rather than `.hic`) or an explicit size cap accepting partial coverage — an 8 GiB cap would reach 43% of the files, 20 GiB 67%, 50 GiB 91%. That is deferred rather than decided. 4DN `.mcool` files are unaffected.
 - **`coordSystem` is omitted for the 21 4DN contact maps that carry no `genomeAssembly`.** It is left out entirely rather than emitted as an empty string: HiGlass matches `coordSystem` against chromosome-info tilesets, so a plausible-looking placeholder risks a silently misaligned track, whereas an absent key makes the client fall back to the `chromsizes` array every tileset info carries. The heatmap still renders.
 - **ICE-balanced heatmaps are unavailable for the 10 flat `.cool` files.** Coarsening runs without `--balance` because balancing a large flat cooler costs minutes to hours; clodius falls back cleanly to unbalanced when no `weight` column exists.
 - **Interchromosomal tiles are served as the concatenated-genome lattice clodius produces**, with no special-casing.

pyproject.toml

@@ -5,13 +5,21 @@ requires = ["setuptools>=64", "setuptools-scm>=8"]
 [project]
 authors = [{ name = "Conrad Bzura", email = "conradbzura@gmail.com" }]
 classifiers = [
-    "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
 ]
 dependencies = [
     "aiohttp",
     "boto3",
+    # Matrix tile serving (issue #82). Was an optional extra gated
+    # `python_version >= '3.12'` because the fork requires 3.12 while
+    # cfdb still supported 3.11; now that the floor is 3.12, it is an
+    # ordinary dependency like any other. `clodius` on public PyPI is a
+    # different codebase from the fork this consumes — the public
+    # releases carry no `tiles_v2` — which is why [tool.uv.sources]
+    # below still pins it to a commit sha on the fork rather than
+    # resolving the name from PyPI.
+    "clodius",
     "click",
     "debugpy",
     "fastapi",
@@ -36,33 +44,16 @@ dynamic = ["version"]
 license = { text = "MIT License" }
 name = "cfdb"
 readme = { file = "README.md", content-type = "text/markdown" }
-requires-python = ">=3.11"
+# Raised from >=3.11 to >=3.12 to carry clodius (issue #82's matrix tile
+# backend) as an unconditional dependency instead of an optional extra —
+# see the `dependencies` comment above. Every image already builds on
+# python:3.12-slim-bookworm, so this does not change the runtime base;
+# it only drops the promise that a 3.11 interpreter can install cfdb at
+# all.
+requires-python = ">=3.12"
 
 [project.optional-dependencies]
-dev = ["allpairspy", "clodius; python_version >= '3.12'", "cloudpickle", "cryptography", "debugpy", "httpx", "hypothesis", "mongomock-motor", "moto[ecs,s3]", "pytest-asyncio", "pytest-mock", "ruff"]
-# Matrix tile serving (issue #82). Deliberately an extra rather than a
-# runtime dependency, for two reasons that both bite if it moves.
-#
-# First, `clodius` on public PyPI is a different codebase from the fork
-# this consumes — the public releases carry no ``tiles_v2`` — so a bare
-# `pip install .` that resolved the name from PyPI would install a
-# package that imports and then fails at the first tile. An extra makes
-# that resolution impossible rather than merely unlikely.
-#
-# Second, the fork requires Python >= 3.12 while cfdb still supports
-# 3.11. An extra alone does not buy that back — uv resolves universally
-# across the whole `requires-python` range and rejects the lock outright
-# — so the dependency carries an explicit `python_version >= '3.12'`
-# marker. On 3.11 the requirement simply is not there, which is what lets
-# the 3.11 support promise survive until the dependency is made
-# unconditional.
-#
-# The consequence is that images built from the Dockerfiles do NOT carry
-# clodius, so the tile routes answer 501 there. See ``cfdb.tilesets.backend``
-# for the single import gate that makes that degradation explicit rather
-# than a partial failure. Promoting this to a real dependency is tracked
-# as follow-up work, gated on the fork being published.
-tiles = ["clodius; python_version >= '3.12'"]
+dev = ["allpairspy", "cloudpickle", "cryptography", "debugpy", "httpx", "hypothesis", "mongomock-motor", "moto[ecs,s3]", "pytest-asyncio", "pytest-mock", "ruff"]
 
 [tool.uv.sources]
 # Pinned to a sha, not a branch: clodius considers ``tiles_v2`` unstable
@@ -72,22 +63,17 @@ tiles = ["clodius; python_version >= '3.12'"]
 # guards, see ``cfdb.tilesets.backend.ClodiusBackend.tile_error``, now
 # bound to ``TilesetError``, and the tile-omission contract change
 # documented on ``cfdb.tilesets.service.TilesetService._read_tiles``) has
-# been merged there. Even though ``main`` now carries ``tiles_v2``, stay
-# pinned to a commit sha rather than a branch reference — a branch
-# reference floats, and what runs must be exactly what was validated.
+# been merged there. Stay pinned to a commit sha rather than a branch
+# reference regardless — a branch reference floats, and what runs must
+# be exactly what was validated.
 #
 # ``main`` does not carry ``clodius.tiles_v2.hic`` — ``load_hic_tileset``
-# degrades to ``TileBackendUnavailable`` for it, same as a build with no
-# clodius at all. That's a no-op for cfdb today: ``.hic`` is refused at
-# the call site regardless (see ``TilesetService._open_hic``) because the
-# module reads local files only, which does not fit this corpus at any
-# disk budget. Serving ``.hic`` at all is tracked as separate follow-up
-# work, not something this pin is expected to grow.
-#
-# The dependency stays an optional extra rather than becoming
-# unconditional because the fork requires Python >= 3.12 while cfdb
-# still supports 3.11; promoting it (and raising the floor) is tracked
-# as follow-up.
+# degrades to ``TileBackendUnavailable`` for it. That's a no-op for cfdb
+# today: ``.hic`` is refused at the call site regardless (see
+# ``TilesetService._open_hic``) because the module reads local files
+# only, which does not fit this corpus at any disk budget. Serving
+# ``.hic`` at all is tracked as separate follow-up work, not something
+# this pin is expected to grow.
 clodius = { git = "https://github.com/abdenlab/clodius", rev = "07f043bea3380bbc38e76f0970d93abed0be4f69" }
 
 [project.scripts]

src/cfdb/api/main.py

@@ -166,11 +166,11 @@ def _build_tileset_service(
 ) -> "TilesetService | None":
     """Build the matrix tile service, or explain why there isn't one.
 
-    Returns ``None`` when the installed build carries no clodius. That is
-    the expected state for images built today — the fork is private, so
-    the Dockerfiles cannot fetch it and it is installed only via the
-    ``tiles`` extra — and it must degrade to a clear 501 on the tile
-    routes rather than taking the whole application down at startup.
+    Returns ``None`` when the installed build carries no clodius. clodius
+    is an ordinary dependency, so this is a broken or incomplete install
+    rather than an expected state — but it must still degrade to a clear
+    501 on the tile routes rather than taking the whole application down
+    at startup.
 
     The service reads ``api.cache`` through a callable rather than
     capturing it, so a lifespan that rebuilds the cache (or a test that