Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] SPARK-33386 - ansi-mode changed ElementAt/Elt/GetArray behavior in Spark 3.1.1 - fallback to cpu #2276

Closed
tgravescs opened this issue Apr 27, 2021 · 0 comments
Closed

[BUG] SPARK-33386 - ansi-mode changed ElementAt/Elt/GetArray behavior in Spark 3.1.1 - fallback to cpu #2276

tgravescs opened this issue Apr 27, 2021 · 0 comments
Assignees
@wjxiz1992
Labels
audit_3.1.0 bug Something isn't working P0 Must have for release

Comments

@tgravescs
Copy link
Collaborator

Describe the bug
Spark made a change to ElementAt/Elt/GetArray with ansi-mode that throws runtime ArrayIndexOutOfBoundsException in Spark 3.1.1 instead of returning NULL. We need to make the same change in our plugin which will be done under #2272. Until that is done we should fall back to the CPU for those when ansi-mode is turned on.

See other details of that change:
[SPARK-33386][SQL] Accessing array elements in ElementAt/Elt/GetArray Item should failed if index is out of bound
https://issues.apache.org/jira/browse/SPARK-33386
apache/spark@6d31daeb6a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@wjxiz1992 wjxiz1992

Labels
audit_3.1.0 bug Something isn't working P0 Must have for release
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@tgravescs @sameerz @wjxiz1992