[conradbzura]

Description

A source is one of two things: a filesystem path, or a factory — a zero-argument callable returning a freshly opened, seekable binary handle. fsspec, smart_open and a plain lambda: open(p, "rb") all produce factories. Adopt that abstraction in the five BBI tilesets, so a bigWig, bigBed or bigInteract can be served from remote storage without first materializing the file on local disk. This is the second of four backend slices under #13; the Source foundation and the HDF5 adoption landed in #18.

A factory rather than an open handle, because the two readers disagree about what they want and only the factory satisfies both. oxbow may reopen its source between queries, so it needs a callable; h5py and pybigtools need a live object, which a factory yields by being called once. A handle cannot be un-consumed, and the repair a caller reaches for — lambda: handle — is a factory that lies, returning the same exhausted handle on every call. Source.coerce therefore refuses a handle at construction, with a message naming the fix.

Extension dispatch is preserved for path-backed sources, and the asymmetry is worth stating precisely: pybigtools.open(<path>) selects a parser from the suffix while pybigtools.open(<handle>) sniffs the bytes, so failing to recognize a suffix costs a different open, and recognizing one commits to the flavour it names. Handing over the path string is the call pybigtools would have received had this module never existed, and the six spellings it accepts take that route. Every other source is handed a live handle — the branch that already existed for a bigInteract and that no caller could reach, because the handle was always opened internally.

The diff reaches past bbi.py for one reason. BBITileset held a second hand-written copy of the open-and-release protocol H5Backed holds for the HDF5 tilesets, and every defect #18's review found in that protocol lived in one copy or the other. The protocol moves to FileBacked in clodius/tiles_v2/_backed.py, and each backend keeps only what differs: which library call opens the target, whether a given source can be handed over as a path, and what makes an opened file the wrong format.

Closes #15

Proposed changes

clodius/tiles_v2/_backed.py — the shared protocol

FileBacked[_R] is generic in the reader a subclass opens, so each backend keeps a precise return type. It owns four things a backend used to own twice.

The lazy open. Constructing a tileset over an unreachable source is not an error; the reader opens on first use. The guard is double-checked under a lock, because concurrent first access without one calls a caller's factory N times and leaves N−1 handles close can never reach, and a caller's handle has no finalizer to fall back on. The reader and its handle are published in a single statement, so close cannot observe one without the other.

The ownership rule. clodius closes what clodius opened and nothing else. A path a reader accepts is handed over as the path string, so only a source the reader cannot take as a path produces a handle, and that handle is the tileset's to close. Neither h5py.File.close() nor BBIReader.close() closes a Python file object it was handed, so releasing a tileset means closing both, reader first, because the reader reads from its handle lazily.

Construction. A subclass does its header read inside _configuring(), a context manager entered after super().__init__(source). Both exits release. On a raise that is cleanup, widened to BaseException so a cancellation landing between the open and the end of construction does not orphan what was opened; a failing close during teardown is suppressed rather than allowed to replace the construction error, since a server renders a TilesetUnavailable as a refusal and an OSError as a 500. On a normal return the release is the registration is free property: a server registers tilesets it may never serve, and for a factory source each one held open is a live remote connection. Measured at 0 descriptors across 50 factory-backed registrations, against 50 before. Two limits bound the claim — a caller who supplies every header-derived parameter skips the only read, which moves the format refusal to the first request; and nothing reaps a reader once it has been used, so steady-state descriptor count equals the number of served tilesets rather than zero, measured at 300 held by 300 served tilesets.

Release. close detaches the reader and the handle under the lock and closes them outside it, because for a factory source the handle is the caller's remote connection and its teardown is a round trip — holding the lock across it stalls every concurrent reopen on a tileset that is already detached, measured at a 355 ms wait behind a handle whose close takes 400 ms. A reader whose own close raises is retained in a quarantine slot rather than dropped, so a later close reclaims it; it is not returned to _reader, because the lazy guard reads that field without the lock and would republish a reader over a handle the same call just closed.

clodius/tiles_v2/bbi.py — the adoption, and four defects

BBITileset derives from FileBacked. Its own reader accessor and release path are deleted, and it implements three hooks: _needs_handle (false for a path whose suffix pybigtools dispatches on), _reader_open, and _validate.

The suffix check was case-folding. pybigtools.open(path) accepts exactly six spellings, case-sensitively, so signal.BW folded to a match, took the path branch, and was refused outright as an invalid file type — while the handle route opens every one of those files. The fast path turned an openable file into an unopenable tileset. The check now compares exact spellings.

close() never closed the reader. It dropped the reference and left the pybigtools.BBIReader to the garbage collector, so a path-backed tileset released its file whenever the collector next ran rather than when it was told to.

A factory-backed tileset could not serve two tiles at once. pybigtools holds a borrow on its reader for the duration of a query, so two threads driving one handle-backed reader raise RuntimeError: Already borrowed — and RuntimeError is not a TileError, so it escaped the per-tile boundary and failed the whole batch as a 500. Measured before the fix: 200 of 400 requests served, 4 of 8 threads dead. Serialization lives in reading(), a context manager that is also the seam a caller reaching past tiles() should use, since the reader it yields carries a constraint the tileset owns. It is entered per reader call rather than per batch: the borrow lasts one query and a batch is mostly work that never touches the reader, so a batch-wide lock put every concurrent request behind all of it — a trivial concurrent tile went from 833 ms at p50 to 7.5 ms. The lock is reentrant, because reading() is published and a caller may nest it.

close() interrupted the read it raced. Because close deliberately does not hold the read lock, it could call BBIReader.close() while a borrow was held, which raises. BBITileset.close now acquires the read lock first. Measured on a factory-backed tileset costing 10 ms a read, closing 50 ms into a request, 5 of 5 trials: before, close raised and the request died with read of closed file; after, close returns and the request is served, for a wait of 37–46 ms. Under page-cached load across 8 threads the wait moved from a median of 0.11 ms to 0.15 ms.

The contract reading() publishes, and its one precondition

Two statements in this area are load-bearing and both are measured rather than argued.

Every read a reading() block starts must also finish inside it. The borrow is released when the query returns, which is what makes one call the right scope — but records() hands back a lazy BigBedEntriesIterator, and advancing one is a read. The eagerness of the two read helpers is therefore load-bearing: fetch writes into a preallocated array and fetch_records materializes its list before the block ends. Measured on a handle-backed reader, five threads by twenty scans, taking the iterator inside the block and draining it after: 76 of 100 queries died with pyo3_runtime.PanicException ... BadData, and all 24 that returned returned truncated lists — 72, 172, 196 and 372 records against a truth of 900, which raises nothing and reaches a client as a well-formed wrong tile. Draining inside the block: 0 of 100, every count exact. Streaming records rather than accumulating them is the obvious next optimization in BBIAnnotationTileset._tile, and it is the one change this contract forbids.

A path-backed reader is exempt from the lock, on a precondition of the process rather than a property of the library. The borrow is on the BBIReader object — a PyO3 RefCell — and not on the Python handle, so a borrow exists to contend on either route: the same eight-thread workload against a path-backed reader in an interpreter where numpy has not been imported refuses 2800 of 3200 queries. With numpy imported, a thread asking to wake 90 ms into a 165 ms path-backed values() regains the interpreter only as that query returns, and the same workload refuses 0 of 3200. bbi.py imports numpy at module scope, so the exemption holds here; nothing enforces it, and the dependency is recorded at the import it rests on. No mechanism linking the two measurements is asserted.

Published surface

Change	Kind	Rationale
BBI*Tileset(path=…) → BBI*Tileset(source=…), annotated SourceLike	Breaking	The parameter accepts a path or a factory, so path named something it no longer is. This reverses an instruction in the issue, for the reason #18's review reversed the identical instruction in #14. Every construction in the tree passes it positionally
BBITileset.file removed; reading() is the only route to a reader	Breaking	The two were both reachable and only one honoured the borrow constraint. The member returned a pybigtools.BBIReader as often as an h5py.File, so file misdescribed it half the time
CoolerTileset.reader → block_reader_cls, typed type[BlockReader]	Breaking	The attribute holds the block-reader class selected by batched, not a reader. A substitute subclasses the public BlockReader, which the docstring now states
BBIInteractionLinksTileset.link_policy is a read-only property	Breaking	A bare attribute let a value assigned after construction skip LinkPolicy coercion, after which the tile loop's catch-all arm served every interaction crossing the tile under the id of whatever was asked for, raising nothing
clodius.core.BinaryHandle added; Source.open returns it	Additive	read, seek, tell, close — exactly what the handle check enforces. Annotated IO[bytes], the method promised two dozen members a conforming handle is never asked for and a minimal one does not have, so with source.open() as h: type-checked and raised
Source.for_reader removed	Breaking	Its last caller went with the extracted protocol: neither reader accepts the callable arm, so both read Source.path instead

Honest scope

A factory-backed tileset's reads do not scale with a worker pool. One tileset holds one reader and reading() serializes every read through it, measured flat from 1 to 8 threads at 177–213 queries/s and 632–648 tiles/s, with 4.2–7.8× recoverable by giving each thread its own reader. The lock is the correct minimal fix — removing it refuses tiles rather than slowing them — and a bounded reader pool is filed against #13. Path-backed tilesets are unaffected.

Three functions on Source — optional, open_text and size — are public, exported, and have no caller in clodius. They belong to the foundation slice and are left to it.

Four review rounds and one blind verification pass are recorded in .sdlc/reviews/issue-#15/. Incidental findings are carried there as deferrals rather than addressed here, since none pertains to this issue.

Test cases

#	Test Suite	Given	When	Then	Coverage Target
1	test_bbi	The same file addressed by path and by a handle factory, for all four concrete tilesets	Every tile in the whole resolution ladder is requested from each	Payloads should match tile for tile, with at least one populated and none an error	Path/factory equivalence
2	test_bbi	The same bigWig under each of the seven display modes	Every tile in the ladder is requested from each source shape	Payloads should match, including the range modes returning several values per bin	Equivalence across modifiers
3	test_bbi	A path-constructed and a factory-constructed tileset	The served info and chromsizes are compared	Both should agree	Equivalence of the info surface
4	test_bbi	A str, bytes, os.PathLike and Source source, for all four tilesets	Each is constructed and asked for a tile	All should serve identically to the str baseline	Every path spelling SourceLike admits
5	test_bbi	An already-open file handle	A tileset is constructed from it	It should raise TypeError naming the callable that would work	Refusal of a live handle
6	test_bbi	A source that is neither a path nor a callable	A tileset is constructed from it	It should raise TypeError	Refusal of an unusable source
7	test_bbi	A factory returning a text handle, and one returning an unseekable handle	A tileset is constructed from each	Each should be refused naming the missing capability, before pybigtools is reached	Capability refusals
8	test_bbi	A factory returning a text handle, with chromsizes supplied so construction defers the open	A tile is served	It should raise then rather than at construction	The deferred-open boundary
9	test_bbi	A source passed as the source keyword	A tileset is constructed	It should serve normally	Keyword reachability of the renamed parameter
10	test_bbi	A factory source and an explicit link_policy	That tile is served	Both should compose, the policy coerced before anything opens	Subclass keyword composition
11	test_bbi	A links tileset constructed with the string "both"	Its link_policy is read	It should be LinkPolicy.BOTH	Coercion through the read-only property
12	test_bbi	A constructed links tileset	A new policy is assigned to link_policy	It should raise AttributeError	The refusal the property was introduced for
13	test_bbi	A path copied under each of six suffixes, recognized and not	The reader is opened	.bw, .bb, .bigWig and .BIGBED should route by their exact spelling and .BW and .bigInteract by handle	The dispatch contract, both arms
14	test_bbi	A factory source whose stringified form ends in a recognized suffix	The reader is opened	It should receive a handle	The check keys off the source, not its repr
15	test_bbi	A tileset whose reader has been opened	The reader is accessed repeatedly	It should open once	The lazy-open memo
16	test_bbi	A factory-backed tileset capped at zero records, and the same capped at one	The whole-genome tile is served	The zero cap should read no bytes and the control should read	The no-I/O short circuit, factory side
17	test_bbi	A tileset closed so the next request opens a cold reader, over both source shapes	A tile is served	The first call on that reader should be the header read, not the tile's own query	The reader is read from before it is published
18	test_bbi	A factory-backed tileset that has opened	It is closed	Both the reader and the handle should be released	Ownership of what the tileset opened
19	test_bbi	A path-backed tileset that has opened	It is closed	Its reader should be closed	The reader leak
20	test_bbi	A bigWig copied to a suffix pybigtools does not dispatch on, so the path still produces a handle	A tile is served and the tileset closed	The descriptor count should rise while open and return to baseline after	The third source shape, where a path owns a handle
21	test_bbi	A tileset already closed once	It is closed again	It should not raise	Idempotent release
22	test_bbi	A tileset that has served a tile and been closed	The same tile is requested again	It should be served identically from a second handle	Reopen after close
23	test_bbi	A tileset used as a context manager	The block exits	Both the reader and the handle should be released	Release through with
24	test_bbi	An alternate ordering the quadtree builder refuses, over a factory and over a path	Construction fails after the open succeeded	What it opened should already be closed	Orphan on construction failure
25	test_bbi	A cold factory-backed tileset and several threads released off a barrier	They all touch the reader for the first time	The factory should be called once and every thread get a working reader	The lazy-open race
26	test_bbi	A cold path-backed tileset and several threads released together	They all touch the reader for the first time	One reader should be published and all threads get a usable one	The same race, path shape
27	test_bbi	A factory-backed tileset over a bigWig large enough that one query yields the GIL, cold and already-opened, with the expected tile taken from a second path-backed tileset so this reader's caches stay cold	Several threads serve the same tile repeatedly	Every payload should match and nothing should be refused	The shared-handle borrow, which is the defect the read lock exists for
28	test_bbi	A factory-backed tileset and several threads scanning records over disjoint ranges, each draining inside its own reading() block	Every scan is counted	None should fail and every count should be exact	The invariant the published contract rests on
29	test_bbi	Many path-backed tilesets over one file, each with its reader opened	They are all closed	The descriptor count should rise while open and return to baseline after	Descriptor release at scale
30	test_bbi	An open tileset over the large fixture, with threads serving tiles and one closing, released together, over both source shapes	The close lands mid-request	close should return, every thread finish, and the tileset remain closeable and reopenable	Liveness of close under load
31	test_bbi	A factory-backed tileset whose handle costs 10 ms a read, with a close landing 50 ms into a request	The close runs	It should return and the interrupted request should be served	The borrow close used to raise, deterministically
32	test_bbi	A path-backed and a factory-backed tileset	Each is rendered	Each should name its class and its source, the factory one as a file-like source	Identification in a traceback
33	test_bbi	A factory-backed tileset and several threads serving through it, cold and already-opened	Concurrent entries into reading() are counted	At most one thread should ever be inside it	The serialization itself, deterministically
34	test_bbi	A factory-backed tileset and one thread entering reading() twice	The nested block runs	It should not deadlock	Reentrancy of the read lock
35	test_bbi	A tileset whose reader raises the borrow error, for both the signal and the annotation read path	A tile is served	It should come back as that tile's refusal, with no file flavour named	The translation that keeps a borrow inside the per-tile boundary
36	test_bbi	A tileset whose reader raises an unrelated RuntimeError	A tile is served	It should propagate rather than become a refusal	The translation is a needle, not a net
37	test_bbi	A bigWig named .bb, opened by path and by factory	Each is constructed	The path route should refuse it and the factory route serve it	The mis-recognition escape hatch the suffix comment documents
38	test_bbi	Each of the five BBI classes	Its constructor signature is compared with the base's	It should accept every parameter the base does, with the same default and kind	Drift between a hand-copied signature and its base
39	test_backed	A tileset whose header read fails, over a reader whose close also fails	It is constructed	It should raise the header error, not the teardown one	Precedence of the construction error over a failing release
40	test_backed	A tileset that constructs cleanly, over a reader whose close fails	It is constructed	It should succeed	The registration release cannot fail a constructor
41	test_backed	An open tileset whose reader's close is made to fail	It is closed, reopened, and closed again once the readers can be closed	The first close should raise, the reader should stay reachable, a fresh reader should be handed back in between, and the later close should reclaim both	Retention of a reader that could not be closed
42	test_backed	A factory-backed source and a subclass whose _needs_handle returns false for it	The tileset is constructed	It should raise TypeError naming the override	The one contract a subclass is told it must not narrow