Skip to content

Commit 7fd81aa

Browse files
dwsmith1983schenksjadivaish
committed
feat: add native Delta Lake scan contrib module (page/row-group pruning + in-scan DVs)
Adds contrib/delta behind a -Pdelta profile: delta-spark keeps all planning (log replay, snapshot, partition pruning); Comet claims the DSv1 scan via a new CometScanRuleExtension SPI and reads data through the same native DataFusion parquet path as CometNativeScanExec, inheriting row-group stats pruning, page-index pruning, and filter pushdown. Deletion vectors are decoded natively into per-file ParquetAccessPlans that intersect with page-index pruning, so deleted rows are skipped in-scan; DV blob and footer fetches run concurrently and footers go through the scan's shared FileMetadataCache (no extra metadata round-trips for DV files). Scalar-subquery data filters are resolved at execution time and pushed to the native reader — a capability stock Spark 3.x lacks entirely. Column mapping name mode, DPP, time travel, checkpoints, schema evolution, and INT96 covered. The `delta` native feature ships in the default set: runtime stays double-gated (contrib jar via ServiceLoader + conf), so it is inert for non-Delta users; roaring is the only net-new default dependency. CI runs the contrib suites on Spark 3.5/4.0/4.1, byte-compiles the dev scripts on Python 3.11-3.14, and keeps the feature-off error path tested. Verified: 39-test differential suite green on Spark 3.5/4.0/4.1; Delta's own suites with Comet injected fully green, 1156/1156 (DeletionVectors, TimeTravel, ColumnMapping, DeleteSQL, UpdateSQL, MergeIntoSQL — one test-only harness patch maps the Comet scan node to its originalPlan for Delta's ScanReportHelper). Local bench (20M rows, release): 1.35x vs stock at 9.4% of bytes on literal bounds; 3.35x on subquery bounds (stock scans 100%, contrib 5%). Supersedes #4366 (delta-kernel-rs contrib) and #4669 (plain-table native scan), deliberately building on both. Co-authored-by: Scott Schenkein <schenksj@yahoo.com> Co-authored-by: Aditya Vaish <adivaish@microsoft.com>
1 parent e095586 commit 7fd81aa

33 files changed

Lines changed: 4555 additions & 260 deletions

‎.github/workflows/ci.yml‎

Lines changed: 12 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -118,6 +118,7 @@ jobs:
118118
iceberg_1_9: ${{ steps.compute.outputs.iceberg_1_9 }}
119119
iceberg_1_10: ${{ steps.compute.outputs.iceberg_1_10 }}
120120
iceberg_1_11: ${{ steps.compute.outputs.iceberg_1_11 }}
121+
delta: ${{ steps.compute.outputs.delta }}
121122
steps:
122123
- uses: actions/checkout@v7
123124
with:
@@ -137,7 +138,7 @@ jobs:
137138
run: |
138139
set -euo pipefail
139140
if [[ "$EVENT_NAME" == "workflow_dispatch" ]]; then
140-
for key in build_linux build_macos benchmark docs spark_3_4 spark_3_5 spark_4_0 spark_4_1 iceberg_1_8 iceberg_1_9 iceberg_1_10 iceberg_1_11; do
141+
for key in build_linux build_macos benchmark docs spark_3_4 spark_3_5 spark_4_0 spark_4_1 iceberg_1_8 iceberg_1_9 iceberg_1_10 iceberg_1_11 delta; do
141142
echo "${key}=true" >> "$GITHUB_OUTPUT"
142143
done
143144
exit 0
@@ -231,6 +232,16 @@ jobs:
231232
spark-full: '3.5.9'
232233
java: 17
233234

235+
delta_contrib:
236+
name: Delta Contrib Tests
237+
needs: changes
238+
if: |
239+
needs.changes.outputs.delta == 'true' &&
240+
(github.event_name == 'push' ||
241+
github.event_name == 'workflow_dispatch' ||
242+
github.event_name == 'pull_request')
243+
uses: ./.github/workflows/delta_contrib_test.yml
244+
234245
spark_4_0:
235246
name: Spark SQL Tests (Spark 4.0)
236247
needs: changes

‎.github/workflows/delta_contrib_test.yml‎

Lines changed: 124 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,124 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: Delta Contrib Tests
19+
20+
# Reusable: invoked by ci.yml. Triggering, path filters, and concurrency
21+
# live in the umbrella workflow.
22+
on:
23+
workflow_call:
24+
25+
env:
26+
RUST_VERSION: stable
27+
RUST_BACKTRACE: 1
28+
# Force GNU ld on Linux: rust-lld cannot resolve -ljvm against the Zulu JDK
29+
# layout installed by setup-java (same rationale as pr_build_linux.yml).
30+
RUSTFLAGS: "-Clink-arg=-fuse-ld=bfd"
31+
32+
jobs:
33+
34+
contrib-delta:
35+
name: Delta contrib (Spark ${{ matrix.profile.spark }})
36+
runs-on: ubuntu-24.04
37+
container:
38+
image: amd64/rust
39+
strategy:
40+
matrix:
41+
profile:
42+
- spark: "3.5"
43+
java_version: "17"
44+
- spark: "4.0"
45+
java_version: "17"
46+
- spark: "4.1"
47+
java_version: "17"
48+
# spark-4.2 is intentionally absent: the contrib profile is dormant
49+
# until a Delta release supports Spark 4.2.
50+
fail-fast: false
51+
steps:
52+
- uses: actions/checkout@v7
53+
54+
- name: Setup Rust & Java toolchain
55+
uses: ./.github/actions/setup-builder
56+
with:
57+
rust-version: ${{ env.RUST_VERSION }}
58+
jdk-version: ${{ matrix.profile.java_version }}
59+
60+
- name: Cache Maven dependencies
61+
uses: actions/cache@v6
62+
with:
63+
path: |
64+
~/.m2/repository
65+
/root/.m2/repository
66+
key: ${{ runner.os }}-java-maven-${{ hashFiles('**/pom.xml') }}-delta-${{ matrix.profile.spark }}
67+
restore-keys: |
68+
${{ runner.os }}-java-maven-
69+
70+
- name: Build native library with the delta feature
71+
run: |
72+
cd native
73+
cargo build --features delta
74+
75+
- name: Install Comet core jars
76+
run: |
77+
./mvnw -B -q -Pspark-${{ matrix.profile.spark }} install -pl common,spark -DskipTests -Dspotless.check.skip=true
78+
79+
- name: Run Delta contrib test suites
80+
run: |
81+
SPARK_HOME=$(pwd) COMET_CONF_DIR=$(pwd)/conf ./mvnw -B -Pspark-${{ matrix.profile.spark }},delta test -pl contrib/delta
82+
83+
# `delta` is in the default feature set, so no regular job builds without it;
84+
# this keeps the feature-off build and its "built without the delta feature"
85+
# error arm (planner.rs cfg(not(feature = "delta"))) from becoming dead code.
86+
feature-off-build:
87+
name: Feature-off native build
88+
runs-on: ubuntu-24.04
89+
container:
90+
image: amd64/rust
91+
steps:
92+
- uses: actions/checkout@v7
93+
94+
- name: Setup Rust & Java toolchain
95+
uses: ./.github/actions/setup-builder
96+
with:
97+
rust-version: ${{ env.RUST_VERSION }}
98+
jdk-version: "17"
99+
100+
- name: Test the delta-off error path
101+
run: |
102+
cd native
103+
cargo test -p datafusion-comet --no-default-features --features hdfs-opendal delta_scan
104+
105+
# The contrib's dev tooling (benchmark and regression-harness scripts) is
106+
# Python; keep it import-clean across currently supported interpreters.
107+
dev-scripts-python:
108+
name: Delta dev scripts (Python ${{ matrix.python-version }})
109+
runs-on: ubuntu-24.04
110+
strategy:
111+
matrix:
112+
python-version: ["3.11", "3.12", "3.13", "3.14"]
113+
fail-fast: false
114+
steps:
115+
- uses: actions/checkout@v7
116+
117+
- name: Setup Python
118+
uses: actions/setup-python@v6
119+
with:
120+
python-version: ${{ matrix.python-version }}
121+
122+
- name: Byte-compile contrib dev scripts
123+
run: |
124+
python -m compileall -q contrib/delta/dev

0 commit comments

Comments
 (0)