fix: ignore NaN sign bits in filters

[lance-gatefixer]

Summary

• ignore NaN sign bits in column and computed comparisons while preserving payloads, stored values, and signed-zero behavior; volatile operands are evaluated once
• keep literal predicates indexable and simplify redundant NaN guards in two-sided bounds, preserving explicit -inf lower bounds and restoring one selective index range
• register the internal normalization function in a copy of the caller's state for scalar and aggregate Substrait decoding
• cover Float16, Float32, and Float64 scan/index behavior, conservative zone-map pruning, the shared memtable planner across three batches, and multi-fragment column comparisons

Root cause

Arrow comparison and scalar-index kernels use total ordering, which places sign-bit-set NaNs below negative infinity while positive NaNs sort above finite values. Lance already rewrote signed-zero literals but did not normalize NaN signs, so logically equivalent NaNs produced different filter results.

The regression tests also pin single evaluation of volatile operands and codec reconstruction. A cold BTree regression with 512 finite values loads one page for [1, 2], instead of the nine pages loaded by the unsimplified range plan. One-sided > and >= retain the additional negative-NaN boundary search.

Validation

• cargo test -p lance-datafusion --features substrait
• cargo test -p lance-index scalar::expression::tests
• cargo test -p lance-index scalar::zonemap::tests::test_nan_zonemap_index
• cargo test -p lance --lib --features slow_tests dataset::mem_wal::memtable::scanner::builder::tests
• cargo test -p lance --test integration_tests --features slow_tests query::primitives::test_nan_sign_is_ignored_in_column_comparisons
• cargo test -p lance --test integration_tests --features slow_tests query::primitives::test_query_float_special_values
• cargo test -p lance --test integration_tests --features slow_tests query::primitives::test_float_zero_predicate_uses_scalar_index
• cargo fmt --all
• cargo clippy --all --tests --benches -- -D warnings
• Python: make build, uv run make lint, and uv run --no-sync pytest python/tests/test_scalar_index.py::test_nan_handling

Fixes #9315

[lance-gatefixer]

Addressed in 3e35bc6: ordered literal comparisons remain scalar-indexable, including computed JSON paths, and mem-WAL combines the NaN guard into one exact range. The additional -inf boundary remains the intentional cost of correcting existing datasets without changing stored NaN bits.

[lance-gatefixer]

Blocked: PR #9324 still requires an approving review before it can merge.

Remote head 08e500df0e66eb65d278493573d89f464864b55c is Ready and mergeable, all live checks pass, and GitHub reports reviewDecision=REVIEW_REQUIRED and mergeStateStatus=BLOCKED; the latest Gatekeeper review is COMMENTED with “approve with a non-blocking risk,” not an approval.

I revalidated every current Review and inline thread: there are no new actionable requests, and the existing App disposition remains later than the only prior author-facing performance concern.

A maintainer can approve the PR; alternatively, Gatekeeper can issue its canonical approval if the documented non-blocking performance risk is acceptable.

[lance-gatefixer]

Blocked: PR #9324 still requires an approving review before it can merge.

Remote head 08e500df0e66eb65d278493573d89f464864b55c is Ready and mergeable, all live checks pass, and GitHub reports reviewDecision=REVIEW_REQUIRED and mergeStateStatus=BLOCKED; the latest Gatekeeper review is COMMENTED with “approve with a non-blocking risk,” not an approval.

I revalidated all current Reviews, inline threads, and later external contributions: there are no new actionable requests, and the existing App disposition remains valid for the prior performance concern.

A maintainer can approve the PR; alternatively, Gatekeeper can issue its canonical approval if the documented non-blocking performance risk is acceptable.

[jonasdedden]

@Xuanwo could you please close this PR, as I continued the bot PR with simplifcations and additional bugfixes in #9628.

[lance-gatekeeper]

⚠️ Gate recommendation: approve with a non-blocking risk.

The three previously reported blockers are fixed. The NaN repair remains read-side and bit-preserving, with reconstructible filters, single evaluation of volatile operands, and selective two-sided index ranges.

The author’s accepted extra −∞ boundary work for one-sided predicates remains non-blocking. Computed comparisons still scan operands for normalization and may allocate when negative NaNs are present; representative workload performance remains unbenchmarked. Performance-sensitive adopters should benchmark their float filters.

[lance-gatefixer]

Blocked: A maintainer must decide whether #9324 should continue or be replaced by #9628.

At remote head 8a69f5316dc4f4c49739da034a30e01a7612450b, the new contributor feedback explicitly asks the bots to stop this PR and continue #9628, citing NaN membership coverage, the separate signed-zero fix, and comparison performance. I verified that #9628 is an open, Ready continuation. This repair remains assigned to #9324 and gatekeeper/fix-9315-1; moving it to another PR or retiring it requires a maintainer decision. The three existing inline dispositions remain valid, and their unresolved threads are unchanged.

Compatibility Tests (GitHub Actions app 15368) still fails on this head: all 30 failures come from the harness pinning lance-namespace>=0.11.1,<0.12 while pylance==14.0.0b10 requires >=0.13.0,<0.14. The dependency conflict was reproduced locally. Current main at 4ac2f7c88be1ea2ab2c0fbb61d02238883fc18b9 now contains the harness correction; it is not yet included in this repair head.

Confirm that #9324 should continue, allowing the upstream harness correction to be integrated and validated. Alternatively, retire #9324 in favor of #9628 and direct further repair work there.

[jonasdedden]

Bots, please focus on #9628 instead and stop working on this PR. Reasons are:

#9628 fixes more

• It closes bug: -0.0 and 0.0 still compare unequal between columns and in array_has #9316 as well as bug: a NaN with its sign bit set sorts below every number in filters #9315. This PR still compares -0.0 and +0.0 by encoding in column comparisons (a = b, a < b, a * -1.0 = a) and in array_has(l, 0.0).
• In fix: ignore zero and NaN sign bits in filters #9628, array_has(l, NaN) ignores the NaN's sign bit. Here it doesn't, even though that case is bug: a NaN with its sign bit set sorts below every number in filters #9315's own scope.
• 14/14 probe cases pass on fix: ignore zero and NaN sign bits in filters #9628 against 8/14 here.

#9628 is faster

• Column comparisons run through one fused function that clears sign bits in registers. This PR scans and copies each operand before comparing.
• On 4M f64 rows, relative to Arrow's plain kernel: x < y costs 1.2–1.35× on fix: ignore zero and NaN sign bits in filters #9628 and 3.5–5.2× here. Literal and range filters cost the same in both.

Same size

• About the same amount of production code (+578/−148 on fix: ignore zero and NaN sign bits in filters #9628 vs +583/−141 here), with one generic comparison function in place of the per-type bit-mask normalizer here.

Nothing from this PR is lost

• The three blockers raised here after the main merge (Substrait decoding, the BETWEEN index range, volatile operands) are all fixed in fix: ignore zero and NaN sign bits in filters #9628. The first two were fixed there before that review.
• This PR's useful tests (memtable rows, the aggregate Substrait decoder, narrow-float index ranges) are ported to fix: ignore zero and NaN sign bits in filters #9628.