Generalise the pigeonhole banding index so a band is an arbitrary set of
columns rather than a contiguous range, and add an entropy-balanced
partition that round-robins columns by per-column conservation
(-ln sum p_i^2). On back-translated (coding) data a contiguous band can
fall entirely inside a conserved region, producing one giant bucket that
dominates both candidate count and the gather/sort cost; balancing spreads
conserved columns across bands so no band is all-conserved.
cluster now defaults to the balanced partition, with conservation estimated
from the first block of input (no stored format needed); --banding
{balanced,contiguous} selects. Output is unchanged (any d+1-band partition
is exact by pigeonhole); verified byte-identical to --no-banding on real
data at d=2,3,5, and the unit tests assert every strategy agrees.
Add a bench-banding subcommand that compares contiguous vs balanced
candidate-set sizes and wall time on a real DB + query set, and reports the
entropy computation cost. On S3.10 (136,770 subjects, 60 bp) balanced is
~1.4-1.7x faster than contiguous for both query and cluster; the entropy
step is ~10 ms, so query can recompute it at load with no DB format change.
Findings and the rejected two-partition variants are written up in
BANDING_BENCHMARK.md.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Idea
For a max Hamming divergence
d, split each sequence intod + 1disjointcontiguous bands. By pigeonhole,
dmismatches cannot touch alld + 1bands,so any centroid within distance
dof a query has at least one band identicalto the query's. Indexing centroids by band value (
band position→band hash→ centroid indices) lets phase 1 compare a sequence only againstcentroids that share a band, instead of all K.
Exactness
This is exact for Hamming distance — it never misses a centroid within
d:dcentroid is guaranteed (pigeonhole) to be in the candidate set.distance check then discards the spurious ones), so they cost time only.
"lowest-index centroid at the minimum distance" tie-break.
d + 1 < len; for largerdan all-differing pair couldshare no band, so phase 1 falls back to the full scan (
WindowSet::nearest).BLOCK_SIZE), so cross-references within a block are handled exactly.Verified byte-for-byte against the v0.8.0 serial binary on 9.4k- and
140k-sequence synthetic sets (clusters + exact duplicates +
-gaps) atd = 0,1,2,3,6,10, spanning single- and many-block runs, and confirmed-t 1==-t 4. Existing unit + cmdline tests pass.Effect
Single-threaded, 140k sequences → 69k clusters at
d = 3: ~58 s (full scan) →~2.5 s (banded). The win is algorithmic (fewer comparisons), so it shows up
even on one core and compounds with the
--threadsparallelism. Benefit growsas K grows and shrinks as
dapproacheslen.Cost / notes
d + 1maps ofband hash → Vec<centroid index>(
u32indices). For very large K and smalldthis is a few extra bytes percentroid per band — worth watching on the 45M run, and an easy future
optimisation (e.g. a single arena instead of many small
Vecs).(no SipHash). No new dependencies.