Skip to content

[WIP] Add Xor Filter to the filters package#740

Draft
freakyzoidberg wants to merge 2 commits into
mainfrom
xorfilter
Draft

[WIP] Add Xor Filter to the filters package#740
freakyzoidberg wants to merge 2 commits into
mainfrom
xorfilter

Conversation

@freakyzoidberg

Copy link
Copy Markdown
Member

Summary

Adds an immutable Xor Filter for probabilistic set membership to a new org.apache.datasketches.filters.xorfilter package, as a smaller/faster alternative to the existing Bloom filter. The API, naming, Javadoc, exception handling, serialization layout, and MemorySegment support mirror BloomFilter.

Implements the algorithm from Graf & Lemire, "Xor Filters: Faster and Smaller Than Bloom and Cuckoo Filters," ACM JEA 2020, with 8-bit and 16-bit fingerprints.

What's included

  • XorFilter — immutable filter: peeling construction, 3-hash membership query, the full typed query(...) suite (long/double/String/byte[]/char[]/ short[]/int[]/long[]/MemorySegment), heapify/wrap, toByteArray/toLongArray, and a BloomFilter-style toString.
  • XorFilterBuilder — accumulates items via a matching typed update(...) suite, de-duplicates, and builds; plus static create(long[]...) factories.
  • package-info.java, XorFilterTest (24 tests), and a standalone XorFilterBenchmark.
  • Family: new XORFILTER(22, "XORFILTER", 3, 3) entry.

Design notes

  • Items are reduced to 64-bit keys once (xxHash64), then mapped to positions and a fingerprint via a MurmurHash3 finalizer keyed on a construction seed that is re-drawn (splitmix64) if a rare 2-core remains, capped at 100 attempts.
  • Fingerprint array capacity is floor(1.23*n) + 32, rounded down to a multiple of 3.
  • Serialization uses a fixed 3-long preamble (the filter has no empty/growable state) followed by the raw fingerprint bytes; layout is documented in an ASCII table in XorFilter.

Testing

  • 24 new unit tests + existing filters/Family tests pass on JDK 25 (no false negatives; typed items; duplicates; empty; determinism; heapify/wrap/toLongArray round-trips for 8- and 16-bit; corruption guards).
  • apache-rat license check passes (0 unapproved).
  • Benchmark confirms the paper's expectations: bits/item = 9.84 (8-bit) / 19.68 (16-bit); observed FPP ~= 1/256 and ~= 1/65536.

Follow-ups (not in this PR)

  • Cross-language (C++/Go) serialization compatibility test, a la BloomFilterCrossLanguageTest.

Hoist the construction-failure throw out of the peeling loop so the loop can terminate through its own bound. Previously the last iteration always exited via break or throw, making the 'attempt < MAX_ITERATIONS' condition provably always-true (CodeQL java/constant-comparison). Behavior is unchanged; the success check now runs after the loop.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants