[conradbzura]

Summary

Add a pure-reader pair of HiGlass-compatible tile endpoints — GET /tileset_info/?d=<uid> and GET /tiles/?d=<uid>.<z>.<x>.<y> — plus a separate POST /tilesets/{dcc}/{local_id} preparation channel that builds the tileset artifact those readers serve from. This lets Gosling's MatrixData (which compiles to HiGlass's native heatmap track, not gosling-track) render Hi-C contact maps against a live tile API instead of a static file.

The tile endpoints never dispatch a workflow — a heatmap track issues thousands of small requests and cannot be handed a job ID to poll — so a contact map with no cached artifact is a plain 404. Building the artifact is the preparation channel's job, mirroring the existing 202-plus-poll pattern used by /data and /index.

Tile reading is backed by a private fork of clodius (abdenlab/clodius, tiles_v2), pinned to a commit sha on its main branch rather than a floating reference, since tiles_v2 is unstable and cfdb is its first consumer. The dependency is an optional extra (uv sync --extra tiles) gated to Python >= 3.12, so cfdb keeps its 3.11 support and images built without the extra answer 501 on the tile routes rather than failing to boot.

Deviation from the issue's open design decision: the issue proposed reading ENCODE .hic in place via hicstraw and HTTP range requests. In practice, clodius's .hic tileset is built on hictkpy, whose entry points take a local path only — there is no URL, HTTP, or S3 surface in the library. Serving .hic would require materializing it first, and the ENCODE corpus makes that infeasible at any disk budget: 3,903 files, ~78 TB total, a median of 10.3 GB, and a largest file of 315 GB, versus a 200 GiB Fargate ephemeral-storage ceiling. .hic is recognized and refused with a specific 501 rather than served; only 4DN .mcool (773 files) and .cool (10 files, coarsened via cooler zoomify first) are servable today. This is documented as a known limitation rather than resolved.

This branch also carries two fixes unrelated to tiling, found while validating the feature against real data:

• The HuBMAP outbound-URL allowlist was missing the DRS metadata broker host, so every HuBMAP file fetch failed with a 502 before ever reaching the asset host.
• HuBMAP's non-public-dataset pruning only matched via the file_in_collection junction table, which HuBMAP's C2M2 export never populates — the materializer's persistent_id fallback join was invisible to it, so files from non-public and dbGaP-controlled-access datasets were leaking into the API stamped data_access_level: public.

And one fix to local dev bootstrap: a clean checkout's database/ directory is gitignored, so the MongoDB image's COPY database/ failed the build outright on a fresh clone (it only appeared to work when a stray file like .DS_Store kept the directory alive). The dump is now mounted read-only at run time instead, and a related getIndexes NamespaceNotFound crash on a from-scratch database is handled.

Closes #82

Proposed changes

Tileset domain layer (src/cfdb/tilesets/)

• backend.py — the single import gate for clodius, so a build without the tiles extra degrades to a clear 501 instead of an ImportError at startup. Classifies clodius's whole deliberate-error hierarchy (TilesetError, not just the narrower TileError) as "render per-tile-id," so a malformed id, an unusable transform, and an out-of-ladder position all become a per-tile {"error": ...} object rather than failing the batch.
• service.py — owns tileset lifetime: a bounded LRU of open h5py handles (reference-counted, so an eviction never closes a handle a reader is still using), a byte-bounded tile cache, and a dedicated thread pool since clodius reads are synchronous h5py + numpy.
• formats.py — classifies a file as mcool, cool, or hic from its filename and file_format, since the C2M2 ontology collapses all HDF5-shaped formats (including AnnData .h5ad) onto one term.
• store.py, wire.py, errors.py — artifact hydration onto local disk, HiGlass tileset_info payload assembly (adding the four fields clodius doesn't model: datatype, name, uuid, coordSystem), and the failure vocabulary that becomes HTTP status codes.

Workflow integration

• A new ArtifactKind.TILESET alongside data/index, sharing the existing content-addressed cache-key derivation (upstream md5, processor version).
• MatrixTilesetProcessor fetches an .mcool byte-identical into the cache, or coarsens a flat .cool with cooler zoomify first (clodius refuses a cooler with no resolutions group). Enforces CFDB_TILESET_MAX_SOURCE_BYTES before a byte is downloaded, including sources whose size is reported as a string.

HTTP endpoints (src/cfdb/api/routers/)

• tiles.py — /tileset_info and /tiles, read-only, capped at 64 repeatable d parameters per request. The response carries exactly one entry per requested id (out-of-ladder and malformed ids render inline as errors rather than being omitted), so a response can safely be built by zipping request against result.
• tilesets.py — POST /tilesets/{dcc}/{local_id} and its /status probe, idempotent (an already-built artifact answers 200 {"ready": true} rather than queuing a no-op job).
• _helpers.py/data.py/index.py — Extract the DCC-normalize/lookup/access-control preamble that /data and /index already shared, so the two new routers reuse it instead of a third and fourth restatement.

Local dev and infra

• Dockerfile.api-tiles and new make api-tiles/make worker-tiles targets layer clodius onto the API image for local testing, since make api's image intentionally carries no clodius while the extra stays optional.
• cloudformation/backend.yml adds ApiEphemeralStorageGiB/TilesetDiskCacheBytes parameters, since h5py needs a tileset artifact on local disk (not just a byte stream) before any tile can be read from it under the S3 profile.

Documentation

• New "Matrix tile serving" README section: the two-channel design, wire protocol and status codes, environment variables, local-dev flow, and documented limitations — including empirical wire-compatibility notes checked against higlass.io and server.gosling-lang.org's live tile servers, and one deliberate divergence from them (an out-of-ladder tile is now a per-tile error object rather than a silently omitted entry).

Test cases

#	Test Suite	Given	When	Then	Coverage Target
1	TestLoadBackend	An environment where clodius is installed	load_backend is called	It returns a backend whose cooler_tileset is clodius's CoolerTileset and whose error attributes are the split clodius exception types	Import gate happy path
2	TestIsTileError	A loaded backend and a MalformedTileId instance	is_tile_error is called	It returns True, even though MalformedTileId is no longer a subclass of the narrower TileError	Per-tile vs 500 classification
3	TestMatrixSourceKind	A file document labelled HDF5 with each supported extension	matrix_source_kind classifies it	It returns the matching MatrixSource, including case-insensitively	Contact-map format discrimination
4	TestMatrixSourceKind	A file document labelled HDF5 with an .h5ad/bare .h5 extension	matrix_source_kind classifies it	It returns None — the HDF5 ontology collision is escaped by filename	Non-contact-map exclusion
5	TestResolution (service)	A contact map with no cached tileset artifact	tileset_info/tiles is called	It raises TilesetNotReady without dispatching anything	Pure-reader contract
6	TestTiles (service)	A batch pairing a valid tile with one past the resolution ladder	tiles is called	Both ids appear in the response — the valid tile returns data, the out-of-ladder one gets a per-tile {"error": ...} object	One-entry-per-id wire contract
7	TestTiles (service)	A tile id naming a balancing column the cooler does not carry, alongside a valid untransformed sibling	tiles is called	Only the bad-modifier id errors — clodius batches per (zoom, modifier), so the valid sibling is unaffected	Finer-grained blast radius
8	TestOpenTilesetLifetime	An open tileset handle evicted past the open cap while a reader is inside checkout	Eviction runs concurrently with the read	The close is deferred until the last reader exits	Reference-counted handle safety
9	TestHydration (store)	A non-local cache artifact	The store opens it	The artifact is downloaded onto local disk once and reused on a repeat request	Single-flight hydration
10	TestEviction (store)	An artifact currently being read	The store evicts to stay under budget	The in-use artifact is never unlinked	Disk-budget safety under live reads
11	TestTilesetInfoPayload (wire)	A file document carrying no genome_assembly	tileset_info_payload builds the response	coordSystem is omitted entirely rather than emitted as an empty string	Chromsizes fallback safety
12	TestSourceSizeGuard (matrix processor)	A source file whose reported size exceeds CFDB_TILESET_MAX_SOURCE_BYTES, including a string-typed size	The processor checks applicability	It refuses before a byte is downloaded	Oversized/mislabelled source guard
13	TestStatusProbe/TestPrepare (router)	The tile backend absent (no clodius)	The status probe or prepare endpoint is called	It raises 501 naming the tile backend and dispatches nothing	Backendless-build degradation
14	TestPrepare (router)	An executor at the active-workflow admission ceiling	prepare_tileset is called	It raises 429 carrying the executor's own Retry-After	Load shedding parity with /data//index
15	TestTilesOverHttp (e2e)	A real cooler built through the matrix processor into a committed cache artifact	/tiles is requested over HTTP	The decoded tile matches a direct cooler read	End-to-end byte fidelity
16	TestEvictionUnderLiveRead (e2e)	Concurrent tile reads against a tileset being evicted from the open-handle cache	Requests run concurrently	No reader sees a closed handle or a segfault-class failure	Concurrency safety under real load
17	TestS3ProfileTileChain	A real mcool committed through the processor into a moto-backed S3Cache	Tiles are requested	The store hydrates once from S3 then serves warm, respecting the disk budget and hydration timeout	S3 profile hydration path
18	TestLifespanProfileMatrix	Every workflow profile (local, S3, mTLS)	The app lifespan boots	The tile service wires against real cache and discovery implementations, not fakes, for each profile	Lifespan wiring across deployment profiles
19	test_physical_names_should_be_environment_scoped (cloudformation)	The CloudFormation templates, including the new ephemeral-storage parameters	Pinned physical names are checked	None is a hard-coded literal that would collide across dev/prod	Environment-scoping invariant
20	TestPruneNonPublicHubmapRawRecords	A non-public HuBMAP dataset whose files carry the dataset DOI as their own persistent_id, with no file_in_collection entry at all	_prune_non_public_hubmap_raw_records is called	The file is deleted via the persistent_id fallback match, even though the file_in_collection-based path finds nothing	Access-control leak fix
21	TestMongoCleanCheckoutBoot	A clean checkout with no database/ dump, built and run through real docker	The MongoDB container boots	It survives with an empty, fully-indexed database rather than crashing on the missing dump	Clean-checkout bootstrap fix
22	TestIndexSpec	The jobs collection's index script running against a namespace that has never been written to	createIndex runs	NamespaceNotFound is swallowed and the index is created rather than crashing the container under set -e	Bootstrap index-creation robustness
23	3.11 CI leg (tile test suite)	An environment where the tiles extra (and therefore clodius/numpy) is not installed, matching Python 3.11 CI	The tile test suite is collected and run	Clodius-dependent tests skip cleanly via pytest.importorskip rather than failing collection or asserting against a backend that cannot exist	Cross-version CI correctness