Repository navigation
Set up the package skeleton, packaging and shared test fixtures — Closes #2 - #7
Draft
conradbzura wants to merge 15 commits into
Draft
conradbzura wants to merge 15 commits into
conradbzura wants to merge 15 commits into
Conversation
conradbzura
force-pushed
the
quadrat-2-package-skeleton
branch
6 times, most recently
from
October 28, 2026 14:25
118d766 to
93206c2
Compare
Somewhere for the ported data path to land. No logic yet, so that a later failure is attributable to the port rather than to the scaffolding it lands on. Follows abdenlab/giql, the nearest sibling and quadrat's own dependency: src layout, hatchling, ruff at line-length 89, four-space indentation, PEP 735 dependency groups. Python floor is 3.11 rather than cherimoya's 3.10, because polars-bio, polars and giql are all gated there and the pileup is not an optional part of what this package is for. Runtime dependencies shear from cherimoya's sixteen to six, with the pileup engine behind a `pileup` extra. Every one of those imports is already lazy inside a function body upstream, so a caller that streams windows over tables it already has never pays for them. Two defects in cherimoya's manifest are corrected rather than carried: pyfaidx is imported by production code but declared only in a dev group, and cherimoya.workers is missing from the explicit package list, so it never ships in a built distribution. Hatchling's src discovery makes the second unrepeatable. The ruff select is pinned rather than left to the default. Ruff 0.17 enables UP, SIM, FURB and RUF by default, and UP032 alone fires on cherimoya's `str.format` style throughout -- so the default would demand a style migration inside the change that moves the code, which is the one thing that makes a port unreviewable. giql resolves from the public commit at the tip of its cherimoya-mvp branch. The datafusion-bio target is not on PyPI and cherimoya pins a path outside its own tree, which a published package cannot inherit. That commit's library code is identical to what cherimoya builds against; the only delta is benchmark material outside src. The fixture layer is the substantive half. Cherimoya has no data-path fixtures in conftest -- every one is module-local, and the FASTA, BAM and BED quintet is duplicated five ways differing only in chromosome sizes and window geometry. Porting that duplication would multiply it, so the builders take the geometry as arguments and the five sets collapse onto one factory. Cut-site builders arrive with the writer they need, in the pileup slice. The cache directory is pointed at a fresh temporary directory at import time, before anything imports the package, because default_cache_dir reads it per call: set later, the first reader would still see the user's own cache, and a surviving entry would satisfy a test written to exercise a cold build.
Review roles, authoring guides and accumulated review documents are local working state, untracked here as they are in the sibling projects. The review history is versioned by its own repository under .sdlc; the guides are config rather than a project artifact.
Hatchling does not read MANIFEST.in -- that is a setuptools file, so its graft and excludes were inert. An sdist built with the file and one built without it came out byte-identical: the test tree was shipping only by way of hatchling's gitignore-based default, never by the deliberate graft the file appeared to express. Naming the contents under the sdist target makes it deliberate, and drops .github and uv.lock from the distribution on the way.
The run was written with a list-slice assignment, so a right-hand side longer than the slice grew the contig rather than being truncated: a 100 bp contig masked over 90 to 120 came out 120 bases long, under a header still declaring 100. Soft masking is immune, since lowercasing preserves the count. A FASTA whose bases and declared lengths disagree is the worst shape this builder can produce. Every later suite's reference comes from here, so the suite would have agreed with itself and with nothing else.
The round-robin branch ran only when the index was divisible by three and then indexed by that same index, so it selected the first contig again whenever the contig count also divided three. Measured over 60 reads: three contigs put all 60 on the first and left two empty, six contigs reached only the first and the fourth. Dividing the index by three before the modulo walks the contigs as intended. The docstring claimed a two-to-one skew toward the first contig. The real ratio is 2k+1 for k contigs, which the corrected text now states along with the short-contig overrun it has always had.
Setting the variable with setdefault honoured an exported value, which is exactly the surviving cache the comment above it rules out: a run by a developer who exports QUADRAT_CACHE_DIR read and wrote their own cache, and the existing cache test failed outright for them. Plain assignment is what the comment already claimed and what the issue asks for. TQDM_DISABLE stays a setdefault. Turning progress bars back on to watch a hang changes nothing a test can observe.
The corpus held the caller's own mapping, and the factory that builds one is session-scoped, so a corpus built from a module constant was that constant. One test mutating what looks like its own attribute would have corrupted the constant for every test after it, in file order, for the rest of the session.
Every later assertion about the pileup, the window cache or the sampler is an assertion about data these builders produced, so a builder that silently writes the wrong thing makes the suite agree with itself and with nothing else. The suite now covers all seven builders -- three had none -- every corpus_factory keyword, Corpus, the environment block and the distribution surface, and a named test fails for each of the five defects fixed in the commits before this one. Split three ways by module under test: the builders, the conftest and the package. The builders file is ordered foundational-first, so a run that stops at the first failure stops at the deepest one rather than at whichever caller happened to come first alphabetically. A pysam fixture replaces the module-level importorskip. That skip aborted collection of all thirteen tests when four needed pysam, and reported it as one line, so a reader could not tell thirteen tests had gone. Without pysam the suite is now 25 passed and 19 skipped, one line each, and the FASTA, BED and interval builders still run -- which is what importing pysam lazily was for.
The fixture layer's prose was written as commentary on the migration -- what the upstream repository carried, which half of the work was larger, what the arithmetic used to do, what the ported suites would do later -- rather than as the contract each member offers a caller. That one habit produced the changelog archaeology, the claims with three and four homes apiece, the counts that were wrong on the commit that introduced them, the enumerations of consumers that do not exist yet, and the implementation mechanics sitting in contract prose. The same orientation is why the guarantees that should have been documented were absent: the N-run clamp, the chromosome-sizes copy, the unbuilt-versus- empty convention and the whole keyword surface of the corpus factory were each pinned by a test and stated in no docstring. One rule applied throughout. Each docstring states what its member does and guarantees; the migration story stays in version control; a claim has exactly one home and every other site points at it. Mechanism that was worth keeping moved under an Implementation notes rubric rather than being deleted. Nothing executable changed. With docstrings stripped, the abstract syntax tree of every touched module is identical to the one before.
conradbzura
force-pushed
the
quadrat-2-package-skeleton
branch
from
October 28, 2026 15:41
93206c2 to
142c822
Compare
write_fasta clamped only its N runs, so a soft mask starting before a contig truncated to an empty slice and masked nothing, while the docstring promised both parameters were clipped to the contig. write_random_bam reserved twice the read length before placing a read, so the last twenty bases of every contig could never be covered and the last forty could never hold a read start. Its own error message justifies one read length, not two, and a later suite expecting signal near a contig boundary would have found a region no read can occupy. aligned_read set the CIGAR only on the placed branch, so an unplaced record discarded the argument without a word: the inferred query length came back empty and the query fell through to the twenty-base fallback. The docstring claimed the opposite, and that claim is now true rather than qualified away. pysam keeps a CIGAR on a record with no reference id and it survives a write, sort, index and read back, and the default 20M infers twenty, so no existing caller sees a change. The docstring also gains the two claims the suite was already relying on: the fallback, which now fires only for a CIGAR consuming no query, and the half-placed record an undeclared contig produces. corpus_factory now rejects header contigs that do not cover the reference at its declared length, which previously produced records the header could not resolve inside a corpus a later suite would assert against, and skips rather than failing with an AttributeError when a caller asks for an alignment without pysam installed. The session cache directory is removed at exit, which the subprocess tests reach and a fixture teardown would not, and each test now reads and writes a cache directory of its own.
Adds the tests the corrected behaviour needs and the ones the suite was missing: a soft mask that starts before its contig, the shortest contig the builder accepts, a read reaching a contig's last base, the contig skew under the condition the docstring states, the header-coverage refusal, the seed on the alignment path, the licence declared once in the built metadata, and the progress-bar setting the conftest docstring contrasts with the cache directory. The unplaced-record test now passes a CIGAR of its own rather than relying on the default, which the twenty-base fallback satisfied either way, so it can tell a CIGAR that was applied from one that was discarded. Collapses the two-contig header into a fixture the sixteen tests that built it inline now request, folds two uv build invocations into one helper that reports the stderr a failed build would otherwise discard, moves the pileup import guard inside the Arrange phase, and renames the tests whose subject is an environment variable or a dunder rather than a method, so each name matches what it exercises. The short-contig refusal is among them: its name said two reads where its body, its docstring, its match string and the builder's guard all say one, and its sibling accepted a contig that is shorter than two reads. Corrects the docstrings that went with them: several described consumers this repository does not contain yet, and several restated a claim their own builder already owns, in one case with the wording already drifted.
The install block presented streaming as current behaviour for a package whose whole surface is a version, and omitted that the pileup extra resolves only with uv until its git-pinned dependency ships to PyPI. The commands the acceptance criteria are written in terms of lived only in the test workflow, which is not where a contributor looks for them.
Four near-duplicates inside the layer whose purpose is de-duplication, all introduced by this branch: bam_placements was record_placements with the until_eof knob dropped, PROJECT_ROOT was written out twice, a module-level strategy named chrom_sizes was defined twice over different domains, and the run-a-child-process block three times. tests/__init__.py already declared itself the home for what the modules share, so that is where they go; the two strategies are renamed for what each generates. FILE_BACKED carries the settings every file-writing property test needs. It also drops the Hypothesis deadline, which those tests were already failing intermittently: a cold first call spends about a second in pysam's sort and index against a 200ms default, and Hypothesis reports that timing as a falsifying example. Four runs at the parent commit failed two; six runs here failed none. The two properties that stay in memory keep the deadline.
write_random_bam documents that a read may start anywhere a record of read_length still ends inside its contig, and both existing properties discarded the position: one asserts the contig set, the other a Counter over contigs. Replacing the span with chrom_sizes[chrom] was caught only by two seed-pinned single-contig examples, so nothing covered an out-of-bounds placement on the multi-contig geometries every consumer of this layer will use. The new property catches that mutation.
write_fasta applies n_runs after soft_mask, so a position covered by both reads as an upper-case N rather than a lower-case base. The suite already depended on it, subtracting the N runs from the soft mask for exactly that reason, so the precedence was pinned in a test while absent from the contract it belongs to. The conftest module docstring rested the cold-build guarantee on the session directory, which no test body ever sees: the per-test fixture is autouse and supersedes it before every test. The session value is what import time and the two re-importing subprocesses observe, and that is now what the module docstring claims, with the per-test guarantee moved to the fixture that delivers it.
conradbzura
force-pushed
the
quadrat-2-package-skeleton
branch
from
October 28, 2026 18:25
142c822 to
f00a79e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stand up the package so the ported data path has somewhere to land: src layout, hatchling, ruff, pytest config, CI, and a shared test fixture layer. No ported logic, so a later failure is attributable to the port rather than to the scaffolding under it.
Follow abdenlab/giql — the nearest sibling and quadrat's own dependency — for the src layout, hatchling, ruff at line-length 89 and PEP 735 dependency groups. Set the Python floor at 3.11 rather than cherimoya's 3.10, because polars-bio, polars and giql are all gated there and the pileup is not an optional part of what this package is for.
Three trade-offs worth noting. The ruff lint select is pinned to
["E4", "E7", "E9", "F", "I"]rather than left to the default: ruff enables UP, SIM, FURB and RUF by default, and UP032 alone fires on thestr.formatstyle used throughout the code this repository is about to receive, so the default would demand a style migration inside the change that moves the code —Iis in anyway, because import order is mechanical even in ported code and is the one style defect a reviewer reliably misses. giql resolves from a pinned public git revision rather than a version specifier, because the datafusion-bio target is not on PyPI; that pin is the one thing blocking publication, and it is why thepileupextra resolves with uv and not yet with pip. And the cut-site table builder and window-geometry arguments that acceptance criterion #4 originally named are relocated to issue #4, the first slice whose tests take either as input, because the on-disk cut-site schema such a builder would have to target is not fixed until #3 lands.Closes #2
Proposed changes
Packaging
Shear the runtime dependencies from cherimoya's sixteen to six and put polars-bio, polars, giql and pyfaidx behind a
pileupextra. Every one of those imports is already lazy inside a function body upstream, so a caller who streams windows over tables they already have never pays for them. State that guarantee over the relationship rather than over the members — nothing the extra installs is imported at package import time — so the extra inpyproject.tomlstays the single enumeration and adding a fifth dependency does not falsify the docstring.Correct two defects in cherimoya's manifest rather than carrying them: pyfaidx is imported by production code but declared only in a dev group, and
cherimoya.workersis missing from the explicit package list so it never ships in a built distribution. Hatchling's src discovery makes the second unrepeatable.Declare the licence once, as a PEP 639 expression with no accompanying classifier, since PyPI rejects a distribution carrying both. Declare the sdist contents where hatchling reads them, with
**on both sides of the checkpoint exclusion: the pattern is gitignore-style, so a single*spans one path segment and would ship every checkpoint directory deeper than one level.Make the pytest configuration enforce itself —
--strict-markersso a misspelt marker is a collection error rather than a test silently neither selected nor deselected, and a deprecation filter scoped to first-party code so a stale API surfaces instead of being silenced along with the third-party noise that motivated the filter.Shared test fixtures
Add
tests/builders.py— plain functions, no fixtures and no state, each rendering one synthetic artifact — and acorpus_factoryfixture over them. Cherimoya has no data-path fixtures in conftest: every one is module-local, and the FASTA, BAM and BED quintet is duplicated five ways differing only in chromosome sizes. Take the geometry as an argument so the five sets collapse onto one factory.State the builders' observable behaviour where a caller reads it rather than where a test happens to assert it.
write_fastaapplies its N runs after its soft mask, so a position covered by both reads as an upper-caseN;aligned_readcarries the CIGAR it is given whether or not the record is placed, falls back to twenty bases only for a CIGAR consuming no query, and produces a half-placed record for a contig the header does not declare. Each of those was already load-bearing somewhere in the suite before it was written down.Import pysam under a guard so the module imports without it, and give the skip two owners that between them cover every path: the suite's
pysamfixture for a test that touches a builder directly, andcorpus_factoryon its own behalf when a call asks for an alignment or a FASTA index. Reject header contigs that do not cover the reference at its declared length, since reads are placed against the reference and a narrower header would otherwise yield records it cannot resolve, inside a corpus a later suite would assert against.Give the cache directory two tiers and document which is in force when. A session directory is assigned at import time, before anything imports the package, because the function that will read it reads it per call: set later, the first reader still sees the user's own cache. That directory is removed at exit rather than on a fixture teardown, which the subprocess tests reach and a fixture would not. During a test body an autouse fixture supersedes it with a directory that test alone can see, so a stale entry from an earlier test cannot satisfy one written to exercise a cold build.
Put what the test modules share in
tests/__init__.pyrather than per module — the geometries, immutably, so a corpus built from one cannot alter the geometry a later corpus is built from; the project root; the BAM reader the assertions compare placements through; the child-process runner that reports the stderr a failed subprocess would otherwise swallow; and the Hypothesis settings every file-writing property test runs under. Those settings carry no deadline, because such a test spends about a second in pysam's sort and index on a cold first call and Hypothesis would report that timing as a falsifying example; the properties that stay in memory keep theirs.Pin the builders' invariants with property tests rather than examples where the domain is wide: that a FASTA holds its declared lengths and masks exactly the intervals it is given, that intervals tile their range, that a header declares any mapping, that a query and its qualities match any CIGAR, that a BAM round-trips any reads in sorted order, that every random read starts far enough from its contig's end to finish inside it, that every contig receives one, and that the surplus skews toward the first.
Repository
Add a README carrying the install shapes, the import guarantee and the commands the acceptance criteria are written in terms of, a
.gitignore, and a test workflow modelled on giql's —permissions: {},persist-credentials: false, a concurrency group, and both actions pinned to full commit SHAs so a re-tagged release cannot execute in CI without a repository change. Sync with--locked, so a green run says the committed lock resolves rather than that some resolution uv wrote and discarded did, and run the bare install as a job of its own, because that is the install the import guarantee is about and syncing without extras would strip the venv the test job needs.Ignore
.sdlc/, which holds local review roles and guides, untracked here as in the sibling projects.Test cases
test_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_buildersNwhere a run covers it and lower case where only a soft mask doestest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_builderstest_buildersTestCorpusTestCorpusTestCorpusTestCorpustest_conftesttest_conftesttest_conftesttest_conftesttest_conftesttest_conftesttest_conftesttest_conftesttest_conftesttest_conftesttest_conftesttest_conftesttest_conftesttest_conftesttest_conftesttest_conftesttest_conftesttest_conftestTestDistributionTestDistribution__all__is enforced rather than documentaryTestDistribution# publicmarker are collected across all of themTestDistributionTestDistributionTestDistributionTestDistribution.gitignore, whose own checkpoint entry would otherwise keep them out of the archive and leave this unable to fail