Skip to content

Fix cosine normalization for small-norm float32 vectors - #688

Open
shoaib0657 wants to merge 1 commit into
nmslib:developfrom
shoaib0657:fix/cosine-small-norm
Open

shoaib0657 wants to merge 1 commit into
nmslib:developfrom
shoaib0657:fix/cosine-small-norm

Conversation

@shoaib0657

@shoaib0657 shoaib0657 commented Oct 2, 2026 •

Copy link
Copy Markdown

Addresses #685.

For a component around 1e-25, squaring it in float underflows to zero. The current normalization then produces cosine distances around -1e10 and +1e10 instead of 0 and 2 for identical and opposite vectors.

This fixes normalization in both Index and BFIndex. It keeps float arithmetic for ordinary vectors and recomputes tiny or overflowing norms in double. Zero vectors keep their existing behavior, and normalized vectors are still stored as float32.

Tests

Added regression tests for tiny and large vectors, different stored/query scales, float boundaries, and zero vectors. The tiny-vector test reproduces the original bug.

Local checks on Ubuntu WSL2 with GCC 13.3, Python 3.12.3, and NumPy 2.5.3:

  • make test: 25 tests passed.
  • make examples: all five examples passed.
  • git diff --check: passed.

Performance

Measured BFIndex.add_items on 50,000 random float32 vectors of dimension 128: seed 685, one thread, one warmup, and 15 timed rounds per version. Hardware: Intel Core i5-11320H under WSL2. Both builds used -O3/C++11 without -march=native; the order of the versions was randomized each round.

Version Median time
Upstream 19.80 ms
This PR 19.59 ms

Insertion time is close to upstream on these ordinary inputs. The timing includes normalization and copying. HNSW build/query performance and workloads dominated by the double fallback were not measured.

Previously saved indexes affected by this bug need to be rebuilt from the original vectors.

@shoaib0657
shoaib0657 marked this pull request as ready for review October 2, 2026 00:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant