Serve bigInteract tilesets directly from remote storage — Closes #120

ENCODE-SUPPLEMENT.md

@@ -55,9 +55,9 @@ ENCODE uses human-readable strings for file formats, assay types, output types,
 
 `cfdb:` marks a term minted here because EDAM has none (verified against OLS4). The alternative — aliasing an unrepresented format onto the nearest EDAM term — is what produced the starch/BED conflation behind #69 and #72: the format becomes indistinguishable from the one it was aliased to, and the processor claiming that term picks it up and mangles it.
 
-`bedpe` and `bigInteract` both pair two loci per record. Routed into the BED tabix pipeline they would be sorted and indexed by the first locus alone, committing a cached artifact that looks successful and is wrong — and the byte-sniff guard from #71 does not catch it, since both are plaintext or gzip. The distinct **name** is what does the work: processor lookup keys on `file_format.name`, and neither name is in any processor's `supported_formats`, so `GET /data/...` streams the raw upstream file rather than a mangled index. Tileset endpoints that understand these formats are planned separately.
+`bedpe` and `bigInteract` both pair two loci per record. Routed into the BED tabix pipeline they would be sorted and indexed by the first locus alone, committing a cached artifact that looks successful and is wrong — and the byte-sniff guard from #71 does not catch it, since both are plaintext or gzip. The distinct **name** is what does the work: processor lookup keys on `file_format.name`, and neither name is in any processor's `supported_formats`, so `GET /data/...` streams the raw upstream file rather than a mangled index.
 
-**`bigInteract` loses a working index in the meantime.** The two formats were not equally broken before this change. A `bedpe` routed through tabix produced an index that was simply wrong — the second mate of every record was unindexed and invisible. A `bigInteract` routed through `bigBedToBed` produced one that was *range-coherent*: the leading three columns are a genuine interval, so the index answered range queries correctly while silently degrading each interaction to a single locus. Re-typing both means `GET /index/...` now returns 404 for either, so `bigInteract` files that previously returned a usable-if-degraded index return nothing at all, and that applies to `bigInteract` already in the experiment corpus as well as to newly ingested annotation files. The tradeoff is accepted deliberately: a 404 states the truth, where the degraded index quietly answered a question the caller did not ask. The tileset work is what restores the capability.
+**`bigInteract` loses a working index, but gets tileset serving in exchange.** The two formats were not equally broken before this change. A `bedpe` routed through tabix produced an index that was simply wrong — the second mate of every record was unindexed and invisible. A `bigInteract` routed through `bigBedToBed` produced one that was *range-coherent*: the leading three columns are a genuine interval, so the index answered range queries correctly while silently degrading each interaction to a single locus. Re-typing both means `GET /index/...` now returns 404 for either, so `bigInteract` files that previously returned a usable-if-degraded index return nothing at all, and that applies to `bigInteract` already in the experiment corpus as well as to newly ingested annotation files. The tradeoff is accepted deliberately: a 404 states the truth, where the degraded index quietly answered a question the caller did not ask. `GET /tileset_info` and `GET /tiles` now serve bigInteract files directly (issue #120) — see the README's "Interaction tile serving (bigInteract)" section — which is the capability `/index`'s degraded answer stood in for. `bedpe` has no such replacement yet; its own tileset endpoints remain planned separately, and its `/index` 404 is still a bare regression with no capability behind it.
 
 This applies to `.bedpe` already in the ENCODE experiment corpus, not only to annotation files — but it reaches no further than ENCODE. `FILE_FORMAT_TO_EDAM` and `get_file_format` have exactly one consumer, the ENCODE transform; 4DN and HuBMAP take `file_format` from their upstream C2M2 datapackage or portal API and never consult the table. A 4DN `.bedpe` whose upstream declares BED therefore still carries `file_format.name == "BED"`, is still claimed by `TabixIntervalProcessor`, and still gets an index built from its first mate. Closing that means routing incoming formats from every DCC through the same table, or refusing a `.bedpe` filename at the processor regardless of declared format; both are follow-up work.
 

README.md

@@ -999,6 +999,27 @@ This is invisible to the client, for two reasons that were verified rather than
 - **The `tileset` artifact for an `.mcool` is a byte-identical copy of the upstream file.** Across all 773 4DN mcools that is a material S3 footprint if every one is ever browsed; the workers stack's `CacheArtifactExpirationDays` (30 days) trims it, at the cost of re-materializing anything revisited after a month.
 - **The local tileset copies are per-API-task.** `DesiredCount` is 1 today; scaling the API horizontally multiplies both the disk footprint and the cold-open cost, and an ALB without session affinity scatters one user's tiles across tasks.
 
+### Interaction tile serving (bigInteract)
+
+`GET /tileset_info` and `GET /tiles` also serve `bigInteract` files — UCSC's bed5+13 format for paired-region interactions (chromatin loops, regulatory element–gene pairs, and the like) — through the same uid scheme as contact maps, but with none of that section's local-materialization machinery: a bigInteract tileset is read straight from the file's upstream URL over HTTP Range requests, via `fsspec`, every time. There is no cache artifact, no `POST /tilesets/{dcc}/{local_id}` preparation step, and no `CFDB_TILESET_DISK_CACHE_BYTES` budget to exhaust — this format never touches `LocalTilesetStore`.
+
+**Two presentations, two uids.** clodius exposes bigInteract three different ways; cfdb resolves two of them and serves both rather than picking one:
+
+| uid | Tileset | `datatype` | Tile id shape |
+|-----|---------|------------|----------------|
+| `{dcc}/{local_id}` | 2D rectangle domains | `2d-rectangle-domains` | `z.x.y` (same convention as a contact map) |
+| `{dcc}/{local_id}:links` | 1D arc/link track | `bedlike` | `z.x` |
+
+clodius's third presentation, `BBIInteractionTileset` ("paired intervals" — the base class the two rows above subclass), is deliberately not resolved: clodius's own docstring for it recommends preferring the two concrete subclasses, and cfdb has no caller for the raw shape.
+
+The `:links` suffix lives only on the wire — it selects which clodius tileset class opens the file, and is stripped before the `(dcc, local_id)` Mongo lookup runs; the file document itself is identical either way. `:` rather than `.` is deliberate: the existing restriction on a `.` inside a uid half (clodius parses `uid.z.x[.y]` by splitting on it) still applies, and `:` never collides with it.
+
+**No `.hic`-style refusal.** `.hic` stays unservable because `hictkpy` has no remote-read entry point at all (see the Limitations above). bigInteract's reader, `pybigtools`, has native HTTP support, which is what makes reading it in place — rather than materializing a local copy — the *default* and only path here, not a future optimization.
+
+**No preparation or readiness probe.** `GET /tilesets/{dcc}/{local_id}/status` and `POST /tilesets/{dcc}/{local_id}` are specific to the contact-map materialization pipeline and do not apply to bigInteract files — `/tileset_info` and `/tiles` are the whole interface.
+
+A bigInteract upstream-resolution failure surfaces through the same per-dataset-vs-infrastructure split every other tileset failure does: a solo request gets a specific HTTP status, and a batch entry renders as an inline `{"error": ...}` object beside its siblings rather than failing the whole request. The status itself has two sources. A DRS lookup that 404s, times out, is forbidden, or errors upstream carries the same 404/403/504/502 that failure would get on `/data` — that part is genuine parity. A malformed or disallowed URL (the SSRF allowlist, a malformed DRS URI, or no usable access method) carries 400, which is cfdb's own choice for the tileset path rather than a mirror of `/data` — `/data` has no 400 case for any of these. A failure in the actual remote read itself (the URL resolves but the fetch then fails, or the bytes aren't a real bigBed) carries 502.
+
 ### Preprocessing & indexing workflow
 
 Many upstream files are not directly consumable by Gosling Designer without preprocessing (e.g., sort+index for BAM, bgzip+tabix for VCF/GFF/BED). When `/data` or `/index` is called for a format that needs preprocessing and the processed artifact is not yet in cache, the API dispatches a workflow and returns `202 Accepted` with a `Location` header pointing to a job status endpoint. A subsequent call, after the workflow completes, streams the processed artifact from cache (with `Range` support). Both endpoints share a single workflow per source file via a Mongo-backed mutex.

pyproject.toml

@@ -23,6 +23,11 @@ dependencies = [
     "click",
     "debugpy",
     "fastapi",
+    # bigInteract tileset serving (issue #120) reads its source directly
+    # over HTTPS Range requests rather than a local copy — `aiohttp`
+    # (already a dependency above) is what fsspec's `http`/`https`
+    # filesystem implementation uses.
+    "fsspec",
     "grpcio>=1.81.1",
     "motor",
     "pydantic>=2,<3",
@@ -58,12 +63,17 @@ dev = ["allpairspy", "cloudpickle", "cryptography", "debugpy", "httpx", "hypothe
 [tool.uv.sources]
 # Pinned to a sha, not a branch: clodius considers ``tiles_v2`` unstable
 # and cfdb is its first consumer, so what runs must be exactly what was
-# validated. This sha is the tip of the fork's ``main`` — the
-# 6-restore-dropped-guards branch (restored ``tiles_v2`` validation
-# guards, see ``cfdb.tilesets.backend.ClodiusBackend.tile_error``, now
-# bound to ``TilesetError``, and the tile-omission contract change
-# documented on ``cfdb.tilesets.service.TilesetService._read_tiles``) has
-# been merged there. Stay pinned to a commit sha rather than a branch
+# validated. This sha is the tip of the fork's ``main`` as of clodius#19
+# ("Accept a file-like source in the bbi tilesets", issue #120's
+# dependency), which lands a shared ``Source`` abstraction (path, or a
+# zero-argument factory returning a fresh seekable binary handle) in
+# ``clodius/tiles_v2/bbi.py`` — what lets cfdb serve a bigInteract tile
+# straight from its upstream URL instead of a local copy. The same
+# abstraction landed for the HDF5-backed tilesets (``cooler``,
+# ``multivec``) in the prior clodius#18; cfdb does not use that side of
+# it yet (``_open_cooler`` still hydrates to local disk — see
+# ``cfdb.tilesets.service``), but the pin brings both along, since they
+# share one commit. Stay pinned to a commit sha rather than a branch
 # reference regardless — a branch reference floats, and what runs must
 # be exactly what was validated.
 #
@@ -74,7 +84,7 @@ dev = ["allpairspy", "cloudpickle", "cryptography", "debugpy", "httpx", "hypothe
 # only, which does not fit this corpus at any disk budget. Serving
 # ``.hic`` at all is tracked as separate follow-up work, not something
 # this pin is expected to grow.
-clodius = { git = "https://github.com/abdenlab/clodius", rev = "07f043bea3380bbc38e76f0970d93abed0be4f69" }
+clodius = { git = "https://github.com/abdenlab/clodius", rev = "610777e89aad82dceaffcbf49ed2b5df1861f6b2" }
 
 [project.scripts]
 cfdb = "cfdb.cli:cli"

src/cfdb/api/routers/tiles.py

@@ -40,6 +40,7 @@
     TilesetHydrationTimeout,
     TilesetTooLarge,
 )
+from cfdb.tilesets.formats import split_bbi_presentation
 from cfdb.tilesets.wire import error_payload
 
 logger = logging.getLogger(__name__)
@@ -92,6 +93,20 @@ def _split_uid(uid: str) -> tuple[str, str]:
     return dcc, local_id
 
 
+def _dcc_and_local_id(uid: str) -> tuple[str, str]:
+    """Split a uid for the file-document lookup, ignoring any presentation.
+
+    A bigInteract uid may carry a ``:links`` suffix (see
+    ``cfdb.tilesets.formats.split_bbi_presentation``) that selects which
+    tileset class serves it — that suffix is meaningless to the Mongo
+    lookup, which is keyed on ``(dcc, local_id)`` alone, so it is
+    stripped here. ``uid`` itself (suffix intact) is still what's passed
+    to ``TilesetService``, which re-derives the presentation on its own.
+    """
+    base_uid, _ = split_bbi_presentation(uid)
+    return _split_uid(base_uid)
+
+
 def _require_service():
     """Return the tile service, or explain why there isn't one.
 
@@ -171,9 +186,12 @@ def _render_dataset_failure(
     """
     _reraise_infrastructure(exc)
     if solo:
-        raise HTTPException(
-            status_code=status.HTTP_404_NOT_FOUND, detail=str(exc)
-        )
+        # TilesetSourceUnavailable carries the specific status /data
+        # already maps the same upstream failure to (404/403/504/502);
+        # every other per-dataset TilesetError has no such attribute and
+        # keeps the plain 404.
+        status_code = getattr(exc, "status_code", status.HTTP_404_NOT_FOUND)
+        raise HTTPException(status_code=status_code, detail=str(exc))
     logger.info(f"Tileset {uid} unavailable: {exc}")
     return error_payload(str(exc))
 
@@ -196,7 +214,7 @@ async def tileset_info(
 
     response: dict[str, Any] = {}
     for uid in uids:
-        dcc, local_id = _split_uid(uid)
+        dcc, local_id = _dcc_and_local_id(uid)
         _, file_doc = await resolve_file_doc(dcc, local_id)
         try:
             response[uid] = await service.tileset_info(uid, file_doc)
@@ -235,7 +253,7 @@ async def tiles(
 
     response: dict[str, Any] = {}
     for uid, uid_tile_ids in grouped.items():
-        dcc, local_id = _split_uid(uid)
+        dcc, local_id = _dcc_and_local_id(uid)
         _, file_doc = await resolve_file_doc(dcc, local_id)
         try:
             response.update(await service.tiles(uid, file_doc, uid_tile_ids))

src/cfdb/tilesets/backend.py

@@ -119,6 +119,57 @@ def load_hic_tileset() -> type:
     return HicTileset
 
 
+def load_bbi_interaction_tilesets() -> tuple[type, type, type, type[Exception]]:
+    """Resolve the bigInteract tileset classes, ``LinkPolicy``, and the
+    pybigtools read-error type.
+
+    Kept separate from :class:`ClodiusBackend`, same reasoning as
+    :func:`load_hic_tileset`: this is a second, independently-gated
+    capability rather than a core field every caller needs. Unlike
+    ``.hic``, there is no permanent refusal behind this gate — pybigtools
+    (the reader these classes use) has native remote-HTTP support, so a
+    clodius new enough to carry them is enough to serve bigInteract tiles
+    straight from the upstream URL. See
+    :meth:`~cfdb.tilesets.service.TilesetService._open_bbi_interaction`.
+
+    ``clodius.tiles_v2.bbi`` also carries a third interaction tileset,
+    ``BBIInteractionTileset`` ("paired intervals") — the base both classes
+    below subclass. It is deliberately not resolved here: clodius's own
+    docstring for it says to prefer the two concrete subclasses, which is
+    what every caller in this codebase does.
+
+    ``pybigtools.BBIReadError`` is resolved here rather than imported
+    directly in ``service.py``, for the same reason the tileset classes
+    are: ``pybigtools`` is pulled in transitively through clodius, not a
+    direct cfdb dependency, so every name from it goes through this one
+    gate.
+
+    Raises:
+        TileBackendUnavailable: when the installed clodius predates
+            clodius#15/#19 and carries no ``clodius.tiles_v2.bbi``
+            interaction tileset classes.
+    """
+    try:
+        import pybigtools
+        from clodius.core.policies import LinkPolicy
+        from clodius.tiles_v2.bbi import (
+            BBIInteraction2DTileset,
+            BBIInteractionLinksTileset,
+        )
+    except ImportError as exc:
+        raise TileBackendUnavailable(
+            "Serving bigInteract tiles requires clodius.tiles_v2.bbi's "
+            "interaction tileset classes, which the installed clodius "
+            "does not provide."
+        ) from exc
+    return (
+        BBIInteraction2DTileset,
+        BBIInteractionLinksTileset,
+        LinkPolicy,
+        pybigtools.BBIReadError,
+    )
+
+
 def is_tile_error(backend: ClodiusBackend, exc: BaseException) -> bool:
     """True when ``exc`` is a clodius per-tile error rather than a bug.
 

src/cfdb/tilesets/errors.py

@@ -65,3 +65,29 @@ class TilesetHydrationTimeout(TilesetError):
     Rendered as 503 with ``Retry-After``: the download may well finish,
     and a retry will find it.
     """
+
+
+class TilesetSourceUnavailable(TilesetError):
+    """Resolving or reading a remote-source tileset's upstream URL failed.
+
+    Only raised by formats served without local materialization (today,
+    bigInteract — see
+    :meth:`~cfdb.tilesets.service.TilesetService._open_bbi_interaction`).
+    Distinct from :class:`TilesetNotReady`: there is no "not yet built"
+    state here, no preparation channel to ask — the failure is in
+    reaching the DCC's own upstream file. ``status_code`` is not one
+    fixed value, and its two sources differ in how closely they track
+    ``/data`` (``cfdb.api.routers.data``): a DRS-resolution failure
+    (object not found, access denied, upstream timeout or error) carries
+    the same 404/403/504/502 that failure maps to on ``/data``, while a
+    malformed or disallowed URL (the SSRF allowlist, a malformed DRS
+    URI, or no usable access method) carries 400 as cfdb's own choice
+    for the tileset path specifically — ``/data`` has no 400 case for
+    any of these, so that one status is not a claim of parity. A batch
+    entry still renders as a plain ``{"error": ...}`` object regardless
+    of which, like every other per-dataset ``TilesetError``.
+    """
+
+    def __init__(self, message: str, *, status_code: int = 502) -> None:
+        super().__init__(message)
+        self.status_code = status_code

src/cfdb/tilesets/formats.py

@@ -94,3 +94,44 @@ def matrix_source_kind(file_meta: dict[str, Any]) -> MatrixSource | None:
 def needs_materialization(file_meta: dict[str, Any]) -> bool:
     """True when this file must be preprocessed before it can be tiled."""
     return matrix_source_kind(file_meta) in MATERIALIZED_SOURCES
+
+
+#: ``file_format.name`` for a bigInteract file — minted rather than an EDAM
+#: term (``services/ontology_mappings.py``), specifically so it is distinct
+#: from plain ``bigBed`` and from this module's own ``HDF5_FORMAT_NAME``.
+#: Unlike :class:`MatrixSource`, a bigInteract file needs no filename-suffix
+#: dance and no materialization: the name alone identifies it, and it is
+#: read in place from its upstream URL — see
+#: :meth:`~cfdb.tilesets.service.TilesetService._open_bbi_interaction`.
+BBI_INTERACTION_FORMAT_NAME = "bigInteract"
+
+
+def is_bbi_interaction_source(file_meta: dict[str, Any]) -> bool:
+    """True when ``file_meta`` is a bigInteract file."""
+    file_format = file_meta.get("file_format")
+    return (
+        isinstance(file_format, dict)
+        and file_format.get("name") == BBI_INTERACTION_FORMAT_NAME
+    )
+
+
+#: Suffix on a tileset uid that selects the 1D arc/link presentation of a
+#: bigInteract file rather than its default 2D rectangle-domain one. ``:``
+#: rather than ``.`` deliberately: a uid half must stay free of ``.``
+#: (clodius parses a tile id by splitting on it), and this never reaches
+#: clodius at all — ``split_bbi_presentation`` strips it before anything
+#: downstream sees the uid.
+LINKS_PRESENTATION_SUFFIX = ":links"
+
+
+def split_bbi_presentation(uid: str) -> tuple[str, str]:
+    """Split a bigInteract uid into its base uid and presentation mode.
+
+    Returns:
+        ``(base_uid, presentation)``, where ``presentation`` is
+        ``"links"`` when ``uid`` ends with :data:`LINKS_PRESENTATION_SUFFIX`
+        and ``"rectangles"`` (the default) otherwise.
+    """
+    if uid.endswith(LINKS_PRESENTATION_SUFFIX):
+        return uid[: -len(LINKS_PRESENTATION_SUFFIX)], "links"
+    return uid, "rectangles"