fix(index): keep zone map zones that may hold a matching NaN

[LuQQiu]

ENT-2893.

This PR fixes incorrect NaN statistics in the writer and missing query results caused by ZoneMap pruning. The fixes apply to both scan-built ZoneMap indexes and data-file write seeds, without changing their file schemas.

Lance follows Arrow Rust's floating-point ordering: negative NaNs sort below non-NaN values, and positive NaNs sort above them.

What was wrong, and what is fixed?

1. Negative NaNs incorrectly changed the maximum

Previously, any NaN made the writer store max = +NaN, even when the NaN was negative.

For a zone containing [-NaN, 5, 8]:

• Before: min = 5, max = +NaN.
• After: min = 5, max = 8.

Negative NaNs are still counted, but no longer change the maximum. Only positive NaNs produce the existing +NaN marker. A zone containing only negative NaNs has null min/max because it has no ordinary values.

2. Range queries could lose NaN rows

For [-NaN, 5, 8], the old reader could skip the zone for x < 0, even though the negative NaN matches under Arrow Rust ordering.

It could also skip a positive-NaN-only zone for x > 100 because the zone had no ordinary minimum.

Fix: range queries keep any zone with nan_count > 0. The existing statistics do not record NaN signs, so keeping these zones is safer than incorrectly skipping matching rows. The actual rows are then filtered normally.

Zones without NaNs still use min/max pruning.

3. NaN query bounds could exclude ordinary values

For example, x > -NaN should match ordinary values. The old reader could instead treat it like x > +NaN and skip the zone.

Fix: the reader distinguishes positive and negative NaN query bounds and checks ordinary values accordingly.

Statistics and compatibility

• min excludes nulls and NaNs. It must not contain NaN for backward compatibility: older readers may incorrectly skip matching rows when min is NaN.
• max excludes negative NaNs; when positive NaNs are present, it retains the existing +NaN marker for backward compatibility with existing readers.
• null_count remains the exact null count.
• nan_count counts both positive and negative NaNs.
• In this PR, the writer counts negative NaNs in memory only to distinguish the signs. A follow-up PR will persist negative_nan_count in the index and write-seed file formats and go through a format vote.

The new reader handles both old and new statistics. Existing files do not need to be rewritten. Older readers still require an upgrade to receive the query fixes.

Tests

Tests cover Float16/32/64, both NaN signs, NaN-only zones, ordinary values, and nulls. They verify the new writer's stored bounds and compare indexed query results with index-disabled scans, before and after write seeds are harvested.

The tradeoff is less range pruning for NaN-containing zones, in exchange for correct results. The follow-up format PR will allow finer pruning of NaN-containing zones using the persisted sign counts.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[LuQQiu]

Follow-up commit dea5fc597 for the review items:

• NaN predicates now reach the index for every float type. The dataset test builds each NaN literal in the column's own type (arrow_cast('NaN', 'Float16' | 'Float32' | 'Float64'), negated for -NaN), so the planner no longer casts the column. assert_representative_queries_use_index checks through explain_plan that ordinary and NaN predicates (<, >, =, IN, BETWEEN, < NaN, > -NaN, = NaN, = -NaN) are planned as a ScalarIndexQuery on the ZoneMap for all three columns, before and after the seeds are harvested. assert_nan_literals_have_expected_sign confirms the typed literals carry the intended sign: = NaN and = -NaN each match exactly the rows written with that canonical bit pattern. The Float16 branches of nan_sign and nan_scalar are now exercised end to end.
• +NaN max described as a marker, not the zone's largest NaN, in the writer and reader comments.
• Format doc (docs/src/format/index/scalar/zonemap.md, no schema change): states that min/max bound the non-null non-NaN values, that max is the canonical +NaN marker exactly when the zone holds a positive NaN while negative NaNs leave the ordinary maximum alone, that older indices wrote the marker for NaNs of either sign, and that range queries keep every zone with nan_count > 0 and leave the exact answer to the row filter.
• Dropped the redundant nan_count local in the seed reader.

On old-reader compatibility: the earlier claim of "no regression" was reasoned from the pruning code and a one-off run of pylance 13.0.0 over a dataset with every NaN/null shape, not proved in general; the examples support it but do not establish more than they cover.

Verified on the build box: cargo test -p lance --lib -- dataset_zone_map_nan and cargo clippy --all --tests -- -D warnings pass.

[github-actions]

Important

Format specification vote

This PR modifies the Lance format specification, so it requires 3 binding +1 votes from PMC members (excluding the proposer), at least one of them on the latest commit, and a minimum 72-hour voting period, weekends excluded, before it can merge. Vote by approving this PR (+1) or requesting changes (−1, a veto). See the voting process.

Approvals carry over across pushes, so a rebase or a typo fix does not send everyone back to re-vote. Whoever approves the latest commit is vouching that nothing substantive has changed since the earlier approvals; if something has, ask for fresh votes.

Status: ❌ Blocked — 0 of 3 required approvals

Approvals	none (0/3)
Latest commit approved by	none — one PMC member must approve the latest commit
Vetoes	none
Voting period	ends Thu 2026-10-15 00:00 UTC (Wed 17:00 PDT)

_{Updated automatically by the format-spec vote gate, which re-checks every 15 minutes — just voted? Re-check now (press Run workflow; leave the input blank to re-check every open format PR). A PMC member may apply the format-waived label to waive the vote for a trivial edit (typo, wording, formatting).}

[lance-gatekeeper]

❌ Gate recommendation: request changes.

The conservative reader fix addresses the NaN pruning errors, and the strengthened typed dataset tests pass. Keep the persisted any-NaN max marker unchanged in this PR so released readers retain sound global bounds.

Please mark this PR with the breaking-change label.

[lance-gatekeeper]

Removing the +NaN max marker for negative-only zones makes released readers report an unsafe global range on newly written indices. v13.0.0's value_range_over detects NaNs through max, without checking nan_count. For [1.0, -NaN], this writer stores min = max = 1.0, so that released method returns [1.0, 1.0], excluding a live value. A mixed-version or rolled-back reader using column_value_range to prune a NaN equality query can silently discard matching rows. The previous writer's marker made the same zone return None.

Keep writing max = +NaN whenever nan_count > 0 in the unchanged index and seed formats. The conservative reader fix can ship with that convention; test released-reader range folding against the newly written statistics.

Executed compatibility reproducer

I added this to the existing scalar::zonemap::tests module. The helper is the v13.0.0 folding method copied verbatim, with only its name changed. The test trains, writes, and loads the current index, verifies the current reader is safe, and checks the old marker as a control.

impl ZoneMapIndex {
    fn gate_released_value_range_over<'a>(
        segments: impl IntoIterator<Item = &'a Self>,
    ) -> Option<(ScalarValue, ScalarValue)> {
        let mut min: Option<&ScalarValue> = None;
        let mut max: Option<&ScalarValue> = None;
        for seg in segments.into_iter() {
            // Nested types have no meaningful ordering
            if seg.data_type.is_nested() {
                return None;
            }
            for zone in seg.zones.iter() {
                // Legacy Decimal zones can contain comparable values even though their
                // extrema were never written, so skipping them would produce a subset.
                if Self::zone_has_missing_extrema(zone) && Self::zone_has_comparable_values(zone) {
                    return None;
                }
                if Self::scalar_is_nan(&zone.max) {
                    return None;
                }
                if Self::scalar_is_finite_bound(&zone.min)
                    && min.is_none_or(|cur| zone.min.partial_cmp(cur).is_some_and(|o| o.is_lt()))
                {
                    min = Some(&zone.min);
                }
                if Self::scalar_is_finite_bound(&zone.max)
                    && max.is_none_or(|cur| zone.max.partial_cmp(cur).is_some_and(|o| o.is_gt()))
                {
                    max = Some(&zone.max);
                }
            }
        }
        Some((min?.clone(), max?.clone()))
    }

    }

    #[tokio::test]
    async fn gate_released_range_preserves_unknown_for_negative_nans() {
        let mut index = train_and_load::<Float32Type>(vec![vec![Some(1.0), Some(-f32::NAN)]]).await;
        assert_eq!(index.value_range(), None, "current reader remains safe");
        assert_eq!(index.zones[0].min, ScalarValue::Float32(Some(1.0)));
        assert_eq!(index.zones[0].max, ScalarValue::Float32(Some(1.0)));
        assert_eq!(index.zones[0].nan_count, 1);
        assert!(ScalarValue::Float32(Some(-f32::NAN)) < index.zones[0].min);

        let saved_max = index.zones[0].max.clone();
        Arc::get_mut(&mut index).unwrap().zones[0].max = ScalarValue::Float32(Some(f32::NAN));
        assert_eq!(ZoneMapIndex::gate_released_value_range_over([index.as_ref()]), None,
            "the released writer's +NaN marker made this unknown");
        Arc::get_mut(&mut index).unwrap().zones[0].max = saved_max;
        assert_eq!(ZoneMapIndex::gate_released_value_range_over([index.as_ref()]), None,
            "released readers must not claim a range that excludes a live negative NaN");
    }

cargo test --profile ci -p lance-index --lib gate_released_range_preserves_unknown_for_negative_nans

The final assertion fails: expected None, observed Some((Float32(1), Float32(1))). This executes the released folding algorithm over the current writer's loaded index, rather than a complete old-version binary.