Skip to content

fix: distinguish reflection failure from absent accessor in IcebergReflection - #5412

Merged
sunchao merged 1 commit into
apache:mainfrom
unikdahal:fix-iceberg-reflection-option-ambiguity
Aug 23, 2026
Merged

fix: distinguish reflection failure from absent accessor in IcebergReflection#5412
sunchao merged 1 commit into
apache:mainfrom
unikdahal:fix-iceberg-reflection-option-ambiguity

Conversation

@unikdahal

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #5258.

Rationale for this change

extractFileLocation, getFileFormat, getEqualityFieldIds collapsed "accessor not
declared" and "accessor threw" into the same None/empty. Live bug: CometScanRule
reads an empty getEqualityFieldIds as "position delete" and goes native, even when
the empty was actually a swallowed failure.

What changes are included in this PR?

Option/empty now means "not declared" only; genuine invoke failures propagate.
Matches findMethod's existing contract, per review comment on the issue. No caller
changes needed at CometScanRule -- its existing outer catch just starts firing
correctly.

How are these changes tested?

New IcebergReflectionSuite cases proving failures propagate instead of collapsing.

@unikdahal

Copy link
Copy Markdown
Contributor Author

@mbutrovich @andygrove this implements the option B you both landed on in #5258, PTAL

@sunchao sunchao left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the distinction between absent accessors and reflection invocation failures, including the callers' fallback handling. I found no P1/P2 issues.

The exact-head ClassLoaders, IcebergReflection, and IcebergReflectionSuite sources compiled against the cached Spark 4.0.4 dependencies, and all 17 reflection tests passed locally. Exact-head CI also passed.

@sunchao
sunchao merged commit bba4744 into apache:main Aug 23, 2026
72 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

IcebergReflection: Option helpers cannot distinguish an absent accessor from a failed reflective call

2 participants