fix(compression): honor configured LZ4 compression

[dentiny]

Hi team, in the current implementation, it reads to me, even when a Binary or Utf8 field explicitly requests lz4, Lance’s variable-width per-value encoding path can use Zstd instead.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[lance-gatekeeper]

❌ Gate recommendation: request changes.

Honoring the configured codec is the right direction, but nullable full-zip writes need a reader-compatible path or an explicit rejection before persistence. The current selection can produce files that fail to read, including with the latest stable reader.

[lance-gatekeeper]

Selecting LZ4 here makes nullable full-zip columns fail to read. Full-zip serialization omits the payload for nulls, and VariableFullZipDecoder::unzip reconstructs them as equal adjacent offsets. CompressedBufferEncoder::per_value_decompress then passes the empty slice to LZ4, which returns LZ4 compressed data too short; Zstd explicitly accepts empty slices. This affects ordinary scans of Utf8, LargeUtf8, Binary, and LargeBinary with nulls, across all three structural encodings.

Handle absent null payloads before invoking LZ4 and keep the existing end-to-end null coverage. Skipping empty slots in per-value decompression restored all 24 round-trip variants locally. The v13.0.0 reader has the same empty-input rejection, so a current-reader fix alone does not preserve stable-format readability: restrict or explicitly reject nullable full-zip LZ4 writes in stable formats until supported released readers can consume them.

Reproducer

In rust/lance-encoding/src/encodings/physical/block.rs, change the existing test_lz4_compress_round_trip expectation from .with_expected_encoding("zstd") to .with_expected_encoding("lz4"), then run:

cargo test -p lance-encoding test_lz4_compress_round_trip

Expected: round trips preserve values and nulls. Observed on this head: all 24 variants fail with LZ4 compressed data too short when the harness reaches null_rate=Some(0.5). The non-null cases pass. Restoring the pre-PR codec selector and original expectation makes all 24 variants pass.