Skip to content

fix: use zero sums for empty AVG partial buffers - #5420

Open
sunchao wants to merge 1 commit into
apache:mainfrom
sunchao:dev/chao/codex/oss-avg-empty-partial-state
Open

fix: use zero sums for empty AVG partial buffers#5420
sunchao wants to merge 1 commit into
apache:mainfrom
sunchao:dev/chao/codex/oss-avg-empty-partial-state

Conversation

@sunchao

@sunchao sunchao commented Aug 22, 2026

Copy link
Copy Markdown
Member

Why are the changes needed?

Closes #5418.

Spark initializes AVG buffers with a zero sum and adds partial sums without coalescing nulls. Comet's empty scalar partial currently exports a null sum instead. In the supported native-partial / Spark-final path, empty scan partitions can therefore turn a nonempty AVG result into NULL.

Decimal AVG must also distinguish an empty buffer from an overflow marker. Changing empty state to zero must not revive an overflowed accumulator or make an all-empty final aggregate divide by zero.

What changes were proposed in this PR?

  • Export a typed zero sum and zero count for empty floating-point and decimal scalar AVG partials, including when no input batch arrives.
  • Preserve decimal overflow markers across state export, partial merges, and later update batches. Handle grouped overflow states with null counts, while keeping empty final results NULL and using errors in ANSI mode versus NULL in legacy/TRY mode for overflow.
  • Add native regressions for empty/all-null inputs, empty/nonempty merges, and overflow-state preservation, plus a multi-partition Spark-final regression for integer and narrow-decimal AVG.

This does not change aggregate mixed-execution eligibility. The separate planner-safety issue is tracked in #5419.

How was this PR tested?

  • Verified fail-before behavior on public main: the new Spark regression returned [NULL, NULL] instead of [2.0, 11.000000]; four of the eight new native tests also failed.
  • cargo test -p datafusion-comet-spark-expr --lib: 646 passed.
  • Spark 4.0.4 / Java 17, CometAggregateSuite filtered with -Dsuites='org.apache.comet.exec.CometAggregateSuite avg': 14 passed, including empty input, mixed execution, decimal overflow, and TRY coverage.
  • Root-reactor Maven package, Spotless, Scala style checks, cargo fmt --all -- --check, and git diff --check passed.
  • Independent standalone-JAR replay on stock Spark 4.0.2: 20/20 synthetic cases match Spark with AQE off/on. The unmodified public-base JAR differs in eight of those cases.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Incorrect AVG result from empty native partial buffers

1 participant