Skip to content

cluster: pidgeonhole and mutltithreading - #51

Merged
wwood merged 9 commits into
mainfrom
pidgeonhole
Aug 11, 2026
Merged

wwood merged 9 commits into
mainfrom
pidgeonhole

Conversation

@AroneyS

@AroneyS AroneyS commented Jun 1, 2026 •

Copy link
Copy Markdown
Collaborator

Idea

For a max Hamming divergence d, split each sequence into d + 1 disjoint
contiguous bands. By pigeonhole, d mismatches cannot touch all d + 1 bands,
so any centroid within distance d of a query has at least one band identical
to the query's. Indexing centroids by band value (band position →
band hash → centroid indices) lets phase 1 compare a sequence only against
centroids that share a band, instead of all K.

Exactness

This is exact for Hamming distance — it never misses a centroid within d:

  • A within-d centroid is guaranteed (pigeonhole) to be in the candidate set.
  • Band-hash collisions can only add candidates, never drop them (a full
    distance check then discards the spurious ones), so they cost time only.
  • Candidates are scanned in ascending index order, preserving the original
    "lowest-index centroid at the minimum distance" tie-break.
  • Used only when d + 1 < len; for larger d an all-differing pair could
    share no band, so phase 1 falls back to the full scan (WindowSet::nearest).
  • In-block new centroids are reconciled by a direct scan in phase 2 (bounded by
    BLOCK_SIZE), so cross-references within a block are handled exactly.

Verified byte-for-byte against the v0.8.0 serial binary on 9.4k- and
140k-sequence synthetic sets (clusters + exact duplicates + - gaps) at
d = 0,1,2,3,6,10, spanning single- and many-block runs, and confirmed
-t 1 == -t 4. Existing unit + cmdline tests pass.

Effect

Single-threaded, 140k sequences → 69k clusters at d = 3: ~58 s (full scan) →
~2.5 s (banded). The win is algorithmic (fewer comparisons), so it shows up
even on one core and compounds with the --threads parallelism. Benefit grows
as K grows and shrinks as d approaches len.

Cost / notes

  • Memory: the index holds d + 1 maps of band hash → Vec<centroid index>
    (u32 indices). For very large K and small d this is a few extra bytes per
    centroid per band — worth watching on the 45M run, and an easy future
    optimisation (e.g. a single arena instead of many small Vecs).
  • Band keys use FNV-1a over the canonical 5-bit codes with an identity hasher
    (no SipHash). No new dependencies.

AroneyS added 2 commits June 1, 2026 11:07
split each seq into d+1 chunks, at least one must match exactly in order to have a max d divergence match
@AroneyS

AroneyS commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator Author

Banding prefilter for query (Win A)

Brings the pigeonhole/banding prune to the query subcommand. Stacks on the
cluster banding work (which is where the shared BandIndex now lives).

Behaviour: a separate path for divergence-bounded queries

  • --max-divergence d set (and d + 1 < window length): build a banding
    index over the subjects at load time and, per query, compare only against
    subjects sharing a band. d + 1 disjoint bands guarantee any subject within
    d shares an identical band, so this is exact.
  • No --max-divergence: kept on the full scan. Without a distance bound the
    nearest hit can be arbitrarily far and share no band, so banding can't prune.
  • --no-banding: forces the full scan even when -d is set — for large d,
    where bands get short and candidate sets blow up. Banding also auto-disables
    when d + 1 >= window length.

Index is built at query startup (HashMap style, band hash -> Vec<subject idx>),
not pre-generated in makedb — that's a separate question; build cost for 500k
is sub-second.

Exactness

Both output branches are reproduced exactly, not approximated:

  • Every within-d subject is guaranteed to be a candidate.
  • Candidates are scanned in (distance, index) order, so the lowest-index
    tie-break is preserved.
  • Output is gated by <= d, and the within-d set is fully captured by the
    candidates, so for both single-best and top-k (incl. --limit-per-sequence)
    the printed set and ordering match the full scan. (The top-k cutoff computed
    over candidates equals the full-scan cutoff whenever it is <= d; above d
    both cap at d.)

Verified byte-for-byte against the v0.8.0 serial binary on a 500k-subject / 60bp
DB across --max-divergence 2/3/4, --max-num-hits 1/3/5/99 and
--limit-per-sequence, and confirmed --no-banding == serial and that the
no--d default is unchanged. Existing cmdline tests use short sequences, so
they land on the full-scan fallback and stay green.

Effect

20k queries vs 500k subjects at d = 3, single thread: ~125 s (full scan) →
~2 s including index build (~63x). This is synthetic uniform data — a near-best
case for pruning. Real marker-gene windows have conserved stretches that inflate
some band buckets, so measure on real data; the win shrinks as d grows toward
the window length (hence --no-banding).

Notes / not included

  • query itself is still single-threaded here; banding is orthogonal to (and
    compounds with) parallelising across queries, which I can add separately.
  • Pre-generating the index in makedb + mmap-loading it is a separate, smaller
    win (sub-second at 500k) discussed earlier; not done here.

@AroneyS
AroneyS marked this pull request as ready for review June 3, 2026 03:44
wwood and others added 6 commits June 4, 2026 22:57
Generalise the pigeonhole banding index so a band is an arbitrary set of
columns rather than a contiguous range, and add an entropy-balanced
partition that round-robins columns by per-column conservation
(-ln sum p_i^2). On back-translated (coding) data a contiguous band can
fall entirely inside a conserved region, producing one giant bucket that
dominates both candidate count and the gather/sort cost; balancing spreads
conserved columns across bands so no band is all-conserved.

cluster now defaults to the balanced partition, with conservation estimated
from the first block of input (no stored format needed); --banding
{balanced,contiguous} selects. Output is unchanged (any d+1-band partition
is exact by pigeonhole); verified byte-identical to --no-banding on real
data at d=2,3,5, and the unit tests assert every strategy agrees.

Add a bench-banding subcommand that compares contiguous vs balanced
candidate-set sizes and wall time on a real DB + query set, and reports the
entropy computation cost. On S3.10 (136,770 subjects, 60 bp) balanced is
~1.4-1.7x faster than contiguous for both query and cluster; the entropy
step is ~10 ms, so query can recompute it at load with no DB format change.
Findings and the rejected two-partition variants are written up in
BANDING_BENCHMARK.md.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
query and cluster now always use the balanced partition
(BandIndex::single_balanced): query recomputes per-column conservation from
the loaded subjects at load (~10 ms, no DB-format change), cluster estimates
it from the first input block. Remove the contiguous option from production
- drop cluster's ClusterBanding enum, the --banding flag, and the strategy
parameter. Output is unchanged: verified byte-identical to --no-banding for
cluster and to the full scan for query on real data at d=2,3.

BandIndex::new (contiguous) is kept only as the bench-banding baseline.

Note next to CURRENT_DB_VERSION that, when the DB version is next bumped, the
per-column conservation weights should be cached in the new format so the
balanced partition need not be recomputed at load.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@wwood
wwood merged commit 258802d into main Aug 11, 2026
7 checks passed
@wwood
wwood deleted the pidgeonhole branch August 11, 2026 07:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants