streaming: record-backend var_base for narrowed/partial-prefix windows (VCF + PGEN)

[d-laub]

Follow-up from issue #277 Wave A Task 4 (annotated output / AnnotatedHaps for the streaming engines, PR TBD).

Context. RecordBackend::generate (src/record_stream/engine.rs) emits annot_v_idxs (dataset-GLOBAL variant ids per output position) by adding a per-window var_base (DecodedWindow::var_base, src/record_stream/transpose.rs) to the window-local ids fill_decoded_window produces. PgenWindowFiller::fill (src/record_stream/pgen.rs) sets this correctly and cheaply from var_start, a value already computed from the .pvar pre-scan's per-contig position table (contig_ranges/contig_pos).

VcfWindowFiller::fill (src/record_stream/vcf.rs) has no equivalent per-contig variant-count/position index — genoray's VcfRecordSource was not found to expose one cheaply — so it hard-codes slot.var_base = 0.

Why this is safe today. Every Wave A VCF annotated fixture (Task 4's test_streaming_annotated_parity.py, Task 5's same-POS fixtures) is single-contig with the region spanning the whole contig, so the window's first decoded variant IS dataset-global variant 0 and var_base = 0 is exact.

The gap. For any config where a VCF-backed StreamingDataset.with_seqs("annotated") window does NOT start at global variant 0 — a multi-contig cohort (2nd+ contig's variants are not global-0-based) or a region that doesn't cover a contig's first variant — annot_v_idxs would be silently WRONG (window-local ids reported as if global).

Action. Add a cheap per-contig variant-count/position table to VcfWindowFiller (analogous to PGEN's .pvar pre-scan — e.g. a lightweight header/index pass at VcfWindowFiller::new, mirroring how sample_indices is already resolved once at construction rather than per-window), then wire slot.var_base the same way PgenWindowFiller does. Add a multi-contig VCF annotated-output parity fixture to exercise the fixed case (mirroring svar1_multicontig_fixture, which already proves the SVAR1 var_base = 0 path is correct for multi-contig).

Add to the StreamingDataset project.

[d-laub]

Scope extended to PGEN (reviewer-confirmed bug found during Task 4 review, fixed on spec/277-output-mode-wave-a).

PgenWindowFiller::fill (src/record_stream/pgen.rs) originally set slot.var_base = var_start as i64, where var_start is the padded, over-inclusive search lower bound used to construct PgenRecordSource (pos.partition_point(...) minus the contig's max-ref-len pad — see the "Narrowed var_start/var_end" section of pgen.rs's module doc). That is not the same thing as the global id of the first variant the window actually keeps: PgenRecordSource::next_record runs a precise anchor-trimmed extent_overlaps filter afterward and can skip leading padded-in candidates whose trimmed extent doesn't reach the window start. When it does, DecodedWindow local column 0 corresponds to global id var_start + skip_count, not var_start — so generate_batch_core's var_base + local silently undercounted every emitted global var_idx by skip_count.

Repro: pgen_snp_ins_del_multi (tests/dataset/conftest.py), region chr1:[71, 200). Variants (global ids): 0=pos29 SNP, 1=pos69 INS (extent [69,70)), 2=pos109 DEL (extent [109,113)), 3/4=pos149 multiallelic-split SNPs. The padded search (pad=4, the DEL's ref len) lands var_start on local index 1 (pos69), but the precise filter skips that candidate since its extent doesn't reach 71 — the window's first kept variant is pos109 (global id 2). Written oracle: var_idxs = [2, 3, 4]. Streamed (buggy): var_idxs = [1, 2, 3].

This was masked in every fixture up to and including Task 4/5 because every annotated fixture's region spans the whole contig (var_start = 0, so skip_count is always 0 too).

Fix landed for Wave A (this branch): PgenWindowFiller::fill no longer sets slot.var_base from var_start at all — it now uses the same var_base = 0 default as VcfWindowFiller, honestly documented as exact only for whole-contig/from-contig-start windows, with the narrowed/partial-prefix/multi-contig case deferred to this issue (same limitation VCF already had). A strict=True xfail (test_annotated_pgen_narrowed_window_var_idxs_gap in tests/dataset/test_streaming_annotated_parity.py) locks the gap so it flips to a failure once fixed here.

Updated action for this issue: a correct fix needs, for both backends, the global id of the first variant a window actually keeps (not a padded/approximate search bound):

• PGEN: extend the existing .pvar pre-scan (contig_ranges/contig_pos) so PgenWindowFiller::fill can derive "first kept variant's global id" directly — e.g. re-running (or reusing) the same extent-overlap trim genoray's PgenRecordSource applies, against the pre-scanned position table, instead of the padded partition_point search alone.
• VCF: still needs the per-contig variant-count/position table described above (no equivalent pre-scan exists yet).

Add both a multi-contig and a narrowed/partial-prefix parity fixture (PGEN's narrowed-window xfail above is a good starting point to convert into a real fixture) to exercise the fixed case for each backend.

[d-laub]

Design correction (2026-07-20) — the scalar var_base scheme is insufficient; genoray must emit per-variant global ids

While turning the Wave B spec into an implementation plan, investigation showed this issue's original framing (fix var_base from var_start, or var_start + skip_count) is fundamentally insufficient, and it's a bigger defect than "narrowed/multi-contig windows."

Root cause: the kept variant set is non-contiguous. OverlapMode::Variant filters on variant extent (POS + ref_consumed), per record in genoray vcf_reader.rs:535-554 (extent_overlaps). A window can exclude an interior variant behind a spanning deletion — e.g. regions [(110,115)] with DEL@100 (extent→120, kept), SNP@105 (extent 106, dropped), SNP@112 (kept) → survivors are global ids {0, 2}, not {0, 1}. Same-POS SNP+DEL mixed-extent ties are a second source. genoray's own SVAR2 test test_gather_keys_excludes_interior_non_overlap_behind_spanning_deletion (spine.rs) asserts exactly [kr(100,0), kr(112,2)].

The record-stream DenseChunk then renumbers survivors to dense local indices 0..n and carries no per-variant global id (types.rs:137-179) — the gap is destroyed inside genoray. A scalar var_base can only reconstruct a contiguous range {base, base+1, …}, which mismatches the written oracle's gapped {base, base+2, …} (_var_ranges.py stores true global ids with gaps). No value of var_base fixes this.

This is a latent bug in already-merged Wave A (#308). with_seqs("annotated") on VCF/PGEN emits annot_v_idxs via var_base + local, so it is silently wrong for any window with a spanning deletion or same-POS tie — even single-contig at window 0, not just the multi-contig case the code comment flags. This issue now also covers that fix.

Decided fix — genoray emits global ids (mirrors SVAR2 pack_vk_src):

• genoray PR: add a per-variant global-id column to DenseChunk, populated by the record source through atomization/exclusion. PGEN is index-addressed (var_start + scan_offset, cheap, tie-safe); VCF is tabix position-addressed and needs a per-contig variant base added at source construction. Merge to genoray main, bump gvl's rev.
• gvl PR-A: add DecodedWindow.global_v_idxs, populate from chunk.global_idx, retire the scalar var_base, re-point annotated (and PR-B1 variants) at per-variant global ids. Parity fixtures for interior exclusion + multi-contig on both VCF and PGEN.

SVAR1 and the SVAR2 query path already carry correct per-variant global ids; only the VCF/PGEN record-stream path is affected.

Spec: docs/superpowers/specs/2026-07-20-streaming-variants-output-wave-b-design.md (PR-310, branch docs/streaming-304-305-spec).