Skip to content

test: fail unexpected vacuous fallback-invariance checks - #5417

Open
sunchao wants to merge 1 commit into
apache:mainfrom
sunchao:dev/chao/codex/fallback-invariance-coverage-guard
Open

test: fail unexpected vacuous fallback-invariance checks#5417
sunchao wants to merge 1 commit into
apache:mainfrom
sunchao:dev/chao/codex/fallback-invariance-coverage-guard

Conversation

@sunchao

@sunchao sunchao commented Aug 22, 2026

Copy link
Copy Markdown
Member

Why are the changes needed?

The suite added in #5329 checks outcomes only after proving that the default and forced-fallback legs exercised different execution paths. However, an unprovable comparison currently becomes SKIPPED-VACUOUS, and the final assertion ignores those entries. The suite can therefore pass with pass=0 fail=0 vacuous=25, or stay green after losing an individual witness such as AddMonths.

Unexpected coverage loss in this fixed regression corpus should fail the test instead of requiring someone to notice a changed counter in the logs. This is test-coverage enforcement, not a product behavior change; other suites also guard the historical AddMonths bug.

Which issue does this PR close?

Follow-up to #5329 and its original issue, #5328. No new issue is closed.

What changes were proposed in this PR?

  • Classify unexpected opaque plans, non-native defaults, and ineffective fallback toggles as FAIL-BIND, which reaches the existing final assertion.
  • Permit only values / StringTranslate / not-native-by-default as the known vacuous exception. If StringTranslate gains native support, it may pass normally; the test does not require a fixed skip count.
  • Keep verdict counters exclusive and update the suite documentation and assertion message to cover binding failures.
  • Add ten regression scenarios in the same suite, covering all binding-gate branches and the expression, section, and reason boundaries of the exception.

Only CometFallbackInvarianceSuite.scala changes. No product code or CI workflow changes.

How was this PR tested?

Rebuilt the native library from this upstream checkout with cargo build --locked --offline, then ran the focused root Maven command on Spark 4.1.3 with JDK 17:

./mvnw -o test -Dtest=none -Dsuites=org.apache.comet.CometFallbackInvarianceSuite

All 6 ScalaTest tests passed, including the ten-scenario guard test in each of the three shuffle/C2R variants. The new guard test also failed in all three variants when compiled against the old implementation, before the binding fix.

Negative controls used the same upstream runtime in isolated JVMs. These counters were observed in every variant:

Configuration Pass Fail Vacuous Result
Healthy baseline 24 0 1 Passed
spark.comet.exec.project.enabled=false 0 24 1 Failed as expected
spark.comet.expression.AddMonths.enabled=false 23 1 1 Failed as expected

Spotless, scalastyle, and git diff --check passed. Local runtime validation is Spark 4.1.3 only; the remaining Spark profiles are left to CI.

@sunchao
sunchao requested review from andygrove and a lite review from Copilot August 22, 2026 19:22

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Strengthens fallback-invariance tests to fail on unexpected binding gaps while preserving the known StringTranslate exception.

Changes:

  • Classifies unexpected gaps as FAIL-BIND.
  • Adds regression coverage for binding branches.
  • Updates documentation and failure messaging.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants