Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support ElementAt #2260

Merged
merged 24 commits into from
May 15, 2021
Merged

Support ElementAt #2260

merged 24 commits into from
May 15, 2021

Conversation

wjxiz1992
Copy link
Collaborator

@wjxiz1992 wjxiz1992 commented Apr 25, 2021 •
edited
Loading

This PR is to close #2060. This gives initial support for element_at.

Note apache/spark#30297 introduce different behaviors for element_at when invalid index or key is given to lhs with ansi mode enabled and disabled.
A follow-up issue is created to track the difference.

For select element_at(Array(1,2,3), 4):

Spark 3.0.2 + spark.sql.ansi.enabled=true:

+-----------------------------+
|element_at(array(1, 2, 3), 4)|
+-----------------------------+
|                         null|
+-----------------------------+

Spark 3.1.1 + spark.sql.ansi.enabled=true:

java.lang.ArrayIndexOutOfBoundsException: Invalid index: 4, numElements: 3

For select element_at(map(1, 'a', 2, 'b'), 3):

Spark 3.0.2 + spark.sql.ansi.enabled=true:

+------------------------------+
|element_at(map(1, a, 2, b), 3)|
+------------------------------+
|                          null|
+------------------------------+

Spark 3.1.1 + spark.sql.ansi.enabled=true:

java.util.NoSuchElementException: Key 3 does not exist.

Signed-off-by: Allen Xu [email protected]

@wjxiz1992 wjxiz1992 self-assigned this Apr 25, 2021
@sameerz sameerz added the feature request New feature or request label Apr 25, 2021
@sameerz sameerz added this to the Apr 26 - May 7 milestone Apr 25, 2021
firestarman
firestarman reviewed Apr 26, 2021
View reviewed changes
integration_tests/src/main/python/array_test.py Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuOverrides.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/complexTypeExtractors.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/complexTypeExtractors.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/complexTypeExtractors.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/complexTypeExtractors.scala Outdated Show resolved Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/complexTypeExtractors.scala Outdated Show resolved Hide resolved
@wjxiz1992 wjxiz1992 linked an issue Apr 26, 2021 that may be closed by this pull request
[FEA] Support element_at function on GPU #2060
Closed
@wjxiz1992 wjxiz1992 mentioned this pull request Apr 27, 2021
Closed
3 tasks
@wjxiz1992 wjxiz1992 marked this pull request as ready for review April 28, 2021 13:07
@firestarman
Copy link
Collaborator

build

firestarman
firestarman previously approved these changes Apr 30, 2021
View reviewed changes
sperlingxx
sperlingxx reviewed May 5, 2021
View reviewed changes
Copy link
Collaborator

@sperlingxx sperlingxx left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found that ElementAt is located at collectionOperations.scala instead of complexTypeExtractors.scala in Spark. Shall we move GpuElementAt into collectionOperations.scala to follow the structure of Spark SQL ?

@revans2
Copy link
Collaborator

revans2 commented May 5, 2021

I found that ElementAt is located at collectionOperations.scala instead of complexTypeExtractors.scala in Spark. Shall we move GpuElementAt into collectionOperations.scala to follow the structure of Spark SQL ?

For most expressions I think it is good to try and mirror where Spark has them so yes.

@wjxiz1992
revert refactor
ae6f8d0 
Signed-off-by: Allen Xu <[email protected]>
@wjxiz1992
Copy link
Collaborator Author

build

@wjxiz1992 wjxiz1992 requested a review from revans2 May 14, 2021 13:48
@wjxiz1992
final fix
03110ab 
Signed-off-by: Allen Xu <[email protected]>
revans2
revans2 previously approved these changes May 14, 2021
View reviewed changes
@wjxiz1992 wjxiz1992 requested a review from firestarman May 15, 2021 05:18
@wjxiz1992
Copy link
Collaborator Author

build

firestarman
firestarman previously approved these changes May 15, 2021
View reviewed changes
@wjxiz1992
Copy link
Collaborator Author

build

@wjxiz1992
Copy link
Collaborator Author

build

@wjxiz1992
indentation
0f55a04 
Signed-off-by: Allen Xu <[email protected]>
@firestarman
Copy link
Collaborator

build

@wjxiz1992 wjxiz1992 merged commit 247b758 into NVIDIA:branch-0.6 May 15, 2021
@wjxiz1992 wjxiz1992 deleted the elementat branch May 15, 2021 15:20
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
@wjxiz1992
Support ElementAt (NVIDIA#2260)
2ffb875 
Support ElementAt.

Signed-off-by: Allen Xu <[email protected]>
nartal1 pushed a commit to nartal1/spark-rapids that referenced this pull request Jun 9, 2021
@wjxiz1992
Support ElementAt (NVIDIA#2260)
44e2619 
Support ElementAt.

Signed-off-by: Allen Xu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sperlingxx sperlingxx sperlingxx left review comments

@firestarman firestarman firestarman approved these changes

@revans2 revans2 revans2 left review comments

Assignees

@wjxiz1992 wjxiz1992

Labels
feature request New feature or request
Projects
None yet
Milestone
May 10 - May 21
Development

Successfully merging this pull request may close these issues.

[FEA] Support element_at function on GPU
5 participants
@wjxiz1992 @firestarman @revans2 @sperlingxx @sameerz