Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Support Empty2Null expression on Spark 3.4.0 #7043

Closed
jlowe opened this issue Nov 10, 2022 · 2 comments
Closed

[FEA] Support Empty2Null expression on Spark 3.4.0 #7043

jlowe opened this issue Nov 10, 2022 · 2 comments
Assignees
Labels
audit_3.4.0 Audit related tasks for 3.4.0 feature request New feature or request Spark 3.4+ Spark 3.4+ issues

Comments

@jlowe
Copy link
Member

jlowe commented Nov 10, 2022

SPARK-40107 added a new expression, Empty2Null, that can appear in Spark 3.4.0 plans. The plugin currently does not support this expression directly, but it does already have the logic to support it in the file format writers that were implicitly performing empty-to-null processing.

@jlowe jlowe added feature request New feature or request ? - Needs Triage Need team to review and classify labels Nov 10, 2022
@sameerz sameerz removed the ? - Needs Triage Need team to review and classify label Nov 16, 2022
@jlowe
Copy link
Member Author

jlowe commented Nov 16, 2022

Note that this is more than just providing an override for the new plan node. This behavior is already being done implicitly in the writer code, as that's how Spark did it before 3.4.x. Therefore on Spark 3.4 we should also stop doing empty2null processing implicitly in the writer, otherwise we're going to waste performance doing a transform that has already been done.

@NVnavkumar
Copy link
Collaborator

Closing since this was implemented in #7167

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
audit_3.4.0 Audit related tasks for 3.4.0 feature request New feature or request Spark 3.4+ Spark 3.4+ issues
Projects
None yet
Development

No branches or pull requests

4 participants