Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/source/dataset.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,5 +168,18 @@ win = ds.with_seqs("variant-windows", gvl.VarWindowOpt(...)).with_output_format(
win.fields["AF"] # same field, alongside win.fields["start"]/["ilen"]
```

Allele fields index per variant, aligned with the scalar fields on the same shared offsets:

```python
rv = ds.with_seqs("variants")[0, 0]

for pos, alt in zip(rv.start[0][0], rv.alt[0][0]):
print(pos, alt) # 10 b'A' / 20 b'GG'
```

This requires **seqpro >= 0.22**. On earlier versions `rv.alt[0][0]` returned the
haplotype's alleles concatenated into one `bytes`, which misaligns against
`rv.start` as soon as a window contains an indel.

See the `genvarloader` skill's `.svar2` `var_fields` section for the field-provenance and
dummy-fill details.
509 changes: 509 additions & 0 deletions docs/superpowers/plans/2026-07-27-seqpro-0.22-bump.md

Large diffs are not rendered by default.

160 changes: 160 additions & 0 deletions docs/superpowers/specs/2026-07-27-seqpro-0.22-bump-design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
# Design: bump seqpro to 0.22.0 (per-variant allele indexing)

**Status:** ✅ approved — 2026-07-27
**Issue:** [#330](https://github.com/mcvickerlab/GenVarLoader/issues/330)
**Upstream:** [ML4GLand/SeqPro#71](https://github.com/ML4GLand/SeqPro/issues/71) (closed 2026-07-27, released in seqpro 0.22.0)
**Branch:** off `main`, PR into `main`

## Problem

On seqpro ≤ 0.21.2, integer-indexing a **string-under-axis** `Ragged` collapsed a whole
group into one concatenated `bytes`, discarding the per-string boundaries that
`str_offsets` already held. Numeric fields sharing the same offsets returned one element
per item, so the two were not shape-consistent:

```python
rv.start[0][h] # -> array([10, 20], dtype=int32) 2 elements, per-variant
rv.ilen[0][h] # -> array([0, 1], dtype=int32) 2 elements, per-variant
rv.alt[0][h] # -> b'AGG' ONE bytes: 'A'+'GG' or 'AG'+'G'?
```

This surfaces through gvl's public API on `RaggedVariants`, where `alt`, `ref`, `start`,
and `ilen` are all built on one shared offsets object (`_share_offsets`,
`_dataset/_rag_variants.py`). The obvious consumer pattern
`zip(rv.start[0][h], rv.alt[0][h])` was silently wrong — and **invisible for SNP-only
data**, because 1-byte alleles make the lengths coincide. It only misbehaves once an indel
appears.

seqpro 0.22.0 fixes it upstream: integer indexing on a string-under-axis `Ragged` now
returns a `Ragged` of strings, and peeling a record row returns opaque-string fields as a
`Ragged` of strings rather than a concatenated `S1` array.

## Scope

Bump the dependency, confirm no fallout, lock the fixed behavior with a gvl-level
regression test, and document the requirement.

**Off `main`, not `streaming`.** The defect is in `RaggedVariants`, which the *written*
`Dataset` read path returns too — `with_seqs("variants")` is not streaming-only. The
`streaming` branch inherits the fix on its next routine `main` merge; no separate
streaming work.

## Changes

### 1. Pins

| File | Line | From | To |
|---|---|---|---|
| `pyproject.toml` | 13 | `"seqpro>=0.20"` | `"seqpro>=0.22"` |
| `pixi.toml` | 104 (`[feature.py310.pypi-dependencies]`) | `seqpro = "==0.21.1"` | `seqpro = "==0.22.0"` |
| `Cargo.toml` | 21 | `seqpro-core = "0.1"` | **unchanged** |

Then regenerate `pixi.lock`.

**`Cargo.toml` stays put.** The 0.22.0 release diff (`0.21.2...0.22.0`) touches only
`python/seqpro/rag/_core.py` plus tests, docs, `pyproject.toml`, and `CHANGELOG.md`. No
Rust changed, so the `seqpro-core` crate needs no bump.

**The floor must move to `>=0.22`.** gvl's public allele-access behavior — and the
regression test below — only hold on ≥ 0.22. A user whose resolver picked 0.21 would
silently get the concatenating behavior back with no error.

Raising the floor also closes a latent inconsistency. The `py310` feature backs the `dev`,
`default`, `py310`, `no-torch`, `notebook`, and `splice` environments and hard-pins seqpro;
`py311`, `py312`, and `py313` declare no seqpro pin at all and inherit the `pyproject.toml`
floor. `pixi.lock` currently holds every environment at 0.21.1, but a lock refresh under
the old `>=0.20` floor would float py311-313 to 0.22 while py310 stayed at 0.21 — two
different allele semantics in one CI matrix.

Compatibility is already satisfied: seqpro 0.22.0 requires `python>=3.10`,
`numpy>=1.26.0`, and `numba>=0.58.1`, against the existing py310 pins of `python 3.10.*`,
`numpy 1.26.*`, and `numba ==0.59.1`.

### 2. Verification

Run the **whole** tree — `pixi run -e dev test` (full pytest tree + `cargo test`), per
CLAUDE.md, not a scoped `tests/dataset` run. This is the real blast-radius measurement.

Alongside it, read the six modules that touch string-`Ragged` internals:

- `_shm_layout.py` (lines 174, 192-193, 695, 746)
- `_dataset/_query.py` (380-386)
- `_dataset/_tracks.py` (133)
- `_dataset/_svar2_haps.py` (992, 1031)
- `_dataset/_flat_variants.py` (101, 119)
- `_dataset/_rag_variants.py`

Static inspection says all six only **construct** string Raggeds
(`Ragged.from_offsets(..., str_offsets=...)` → `.to_strings()`) or access fields **by
name** (`rv[fname]`, `f._rl.str_offsets`). Neither path changed in 0.22.0, so this is
expected to come back clean — but it must be confirmed, not assumed. `_shm_layout.py` is
the one to watch: it round-trips allele fields through shared memory for the
`double_buffered` loader, decomposing them into `outer_offsets` / `inner_offsets` /
`leaf_data` and rebuilding via `to_strings()`.

If the suite does surface fallout, fix it in this branch; the fix is in scope.

### 3. Regression test

The suite currently contains **no** test that integer-indexes an allele field. That gap is
exactly why #330 went unnoticed, so closing it is part of the fix rather than a nicety.

Add a test in `tests/unit/dataset/` (beside `test_flat_variants_type.py`) that builds a
`RaggedVariants` whose group holds a **multi-byte allele** — e.g. alts `b"A"` and `b"GG"`
against starts `[10, 20]` — and asserts per-variant alignment:

- indexing the allele field for a haplotype yields **two** alleles, matching the two
entries `start`/`ilen` yield for that same haplotype
- those alleles equal `b"A"` and `b"GG"` respectively

The exact accessor and comparison form are settled during implementation against the real
0.22.0 return type (a `Ragged` of strings, whose element access this spec deliberately does
not guess at). What the test must pin down is the *alignment property*, not a particular
call.

The multi-byte allele is the entire point. With SNPs only, every allele is 1 byte, the
counts coincide, and the test passes on both 0.21 and 0.22 — gating nothing.

### 4. Docs

A user-visible behavior change, so CLAUDE.md's docs-audit gate applies.

Nothing in the current docs is *false*: no prose demonstrates integer allele indexing, and
`docs/source/dataset.md:164-165` shows only field access (`rv["AF"]`). The work is
therefore additive — a short "reading variants per-variant" note in
`skills/genvarloader/SKILL.md` and in `docs/source/dataset.md` beside that existing
example, showing per-variant allele access and stating the `seqpro>=0.22` requirement.

`docs/source/api.md` is unchanged — no symbol is added to or removed from `__all__`.

### 5. Versioning and close-out

Land as a conventional commit `fix!:` with a `BREAKING CHANGE:` footer, so commitizen cuts
**0.40.2 → 0.41.0**.

`RaggedVariants.alt[i]` / `ref[i]` change observable return type for gvl users, which is
breaking regardless of arriving via a dependency. The counter-argument — that it is "just a
dep bump" of behavior nobody could sensibly rely on — was considered and rejected: silent
type changes in a public return value are exactly what a major/minor signal exists for.

The PR closes #330 and references SeqPro#71.

## Out of scope

- **#328** (`var_fields` ride-alongs for `variant-windows`) and **#329** (SVAR1 per-call
FORMAT copy) — the other two Wave B follow-ups. Streaming-board work, unrelated to this
bump.
- Any change to `Cargo.toml` / `seqpro-core`.
- Adding seqpro pins to the `py311`/`py312`/`py313` features. Raising the shared floor is
sufficient; per-feature pins would duplicate the constraint in four places.

## Success criteria

1. `pixi.lock` resolves seqpro 0.22.0 in every environment.
2. Full `pixi run -e dev test` green (pytest tree + cargo).
3. The new regression test passes on 0.22.0, and is confirmed **once** during
implementation to fail on 0.21.1 — otherwise it does not actually gate the fix. (It is
not kept runnable against 0.21.1 after the floor moves.)
4. SKILL.md and `docs/source/dataset.md` document per-variant allele access and the
`seqpro>=0.22` requirement.
5. PR into `main` closes #330.
Loading
Loading