feat: project Spark 4 VARIANT columns in native Parquet scans [Splitting] - #5407

Draft

peterxcli wants to merge 29 commits into

apache:mainfrom

peterxcli:feat/native-variant-proj

Commits on Aug 21, 2026

feat: project Spark 4 VARIANT columns in native Parquet scans
peterxcli
committed
45a0ed4

Commits on Aug 22, 2026

review
peterxcli
committed
784c316

Commits on Aug 23, 2026

review
peterxcli
committed
c355fef
add link about spark and arrow-rs issue that need to be fixed so we can get cleaner code
peterxcli
committed
c556a52
update spark issue link
peterxcli
committed
9742a7b

Commits on Aug 24, 2026

Merge branch 'main' into feat/native-variant-proj
peterxcli
authored
0a079de
widen unsigned shredded field
peterxcli
committed
33e513c

Commits on Aug 25, 2026

fix: preserve Spark Variant compatibility when unshredding
peterxcli
committed
ed39c4c

Commits on Aug 26, 2026

Merge remote-tracking branch 'upstream/main' into feat/native-variant-proj
peterxcli
committed
8fb17b1
fix: normalize shredded Variant types before unshredding
peterxcli
committed
360a653
refactor: isolate Parquet Variant normalization
peterxcli
committed
a800bc8
fix: preserve strict Variant Parquet validation
peterxcli
committed
ba37a68
review
peterxcli
committed
1e832c1
fix: normalize remaining Variant scan encodings
peterxcli
committed
a784bea

Commits on Aug 27, 2026

fix: align native Variant Parquet scan semantics
peterxcli
committed
389b134
style: remove redundant interpolation
peterxcli
committed
43e7e17
fix: preserve Spark semantics for Variant scans
peterxcli
committed
23bfd83
fix: preserve Parquet Variant physical semantics
peterxcli
committed
22f588a
Merge branch 'main' of https://github.com/apache/datafusion-comet into feat/native-variant-proj
peterxcli
committed
c2778b7
fix: narrow Variant Unicode scan fallback
peterxcli
committed
a352f46

Commits on Aug 28, 2026

test: cover entirely null Variant columns end to end

peterxcli
and
claude
committed
54cfd1f
fix: name pushVariantIntoScan in the Variant scan fallback reason

peterxcli
and
claude
committed
0cc63a7
docs: apply prettier formatting to datatypes table

peterxcli
and
claude
committed
ba7677c
fix: fail closed on Unicode-folded physical names in Variant scans

peterxcli
and
claude
committed
e78da57
fix: restrict the Unicode fold guard to referenced scan columns

peterxcli
and
claude
committed
53c58be
fix: exempt ID-resolved columns from name-duplicate validation

peterxcli
and
claude
committed
e40b939
fix: match Spark's fold-ambiguity and field-ID shadowing semantics

peterxcli
and
claude
committed
4925cc2

Commits on Aug 29, 2026

Merge branch 'main' into feat/native-variant-proj
peterxcli
authored
429d598
cargo fmt
peterxcli
committed
519b281