Batching support for ROW-based FIRST() window function
#9489
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This commit adds support for `FIRST()` window functions, running in a ROWS context. This does not currently support ignore/include nulls. Signed-off-by: MithunR <[email protected]>
Plus, tests for partitioned/unpartitioned, and keep/ignore nulls.
Also removed unused import.
|
Build |
mythrocks
added
improve
reliability
revans2
previously approved these changes
Oct 23, 2023
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuWindowExec.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: MithunR <[email protected]>
Agreed. The latest commit addresses that. The snag was that
I've filed #9520. I'd be happy to tackle it right after the |
|
Build |
revans2
reviewed
Oct 24, 2023
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuWindowExec.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuWindowExec.scala
Outdated
Show resolved
Hide resolved
sql-plugin/src/main/scala/org/apache/spark/sql/rapids/aggregate/aggregateFunctions.scala
Show resolved
Hide resolved
|
Build |
|
The failure last time around was a Parquet test for timestamp push down. |
That is a known issue and the test has been disabled while CUDF tries to fix it. Please upmerge and then rerun your tests. |
revans2
previously approved these changes
Oct 25, 2023
|
Build |
|
This test appears to be failing because the |
|
Build |
revans2
approved these changes
Oct 26, 2023
|
build |
|
I've merged this change. Thank you for the reviews. Focusing on #9544 now, for |
mythrocks
added a commit
to mythrocks/spark-rapids
that referenced
this pull request
Oct 30, 2023
Fixes NVIDIA#9569. NVIDIA#9489 added `NTH_VALUE()` tests with option to `IGNORE NULLS`, but mistakenly enabled `IGNORE NULLS` for Spark versions prior to `3.2.1`. This commit restricts tests for `IGNORE NULLS` to only Spark versions exceeding `3.1.x`, where the feature is available.
mythrocks
added a commit
to mythrocks/spark-rapids
that referenced
this pull request
Oct 30, 2023
Fixes NVIDIA#9569. NVIDIA#9489 added `NTH_VALUE()` tests with option to `IGNORE NULLS`, but mistakenly enabled `IGNORE NULLS` for Spark versions prior to `3.2.1`. This commit restricts tests for `IGNORE NULLS` to only Spark versions exceeding `3.1.x`, where the feature is available. Signed-off-by: MithunR <[email protected]>
pxLi
pushed a commit
that referenced
this pull request
Oct 31, 2023
Fixes #9569. #9489 added `NTH_VALUE()` tests with option to `IGNORE NULLS`, but mistakenly enabled `IGNORE NULLS` for Spark versions prior to `3.2.1`. This commit restricts tests for `IGNORE NULLS` to only Spark versions exceeding `3.1.x`, where the feature is available. Signed-off-by: MithunR <[email protected]>
mythrocks
added a commit
to mythrocks/spark-rapids
that referenced
this pull request
Nov 3, 2023
Fixes NVIDIA#9520. Fixes NVIDIA#9299. This is a followup to the changes in NVIDIA#9489, which adds the running window optimization to the `FIRST()` window fuction. This commit adds running window support for the `LAST()` window function, with support for both `ROWS` and `RANGE` based window specifications. This change should allow the `LAST()` aggregation to run in multiple batches if the window is `[UNBOUNDED PRECEDING, CURRENT ROW`]. This will allow for much larger window and group sizes, because the group no longer needs to fit in GPU memory. This should help mitigate out-of-memory errors with `LAST()` window aggregations. Signed-off-by: MithunR <[email protected]>
mythrocks
added a commit
that referenced
this pull request
Nov 6, 2023
* Running window optimization for LAST() Fixes #9520. Fixes #9299. This is a followup to the changes in #9489, which adds the running window optimization to the `FIRST()` window fuction. This commit adds running window support for the `LAST()` window function, with support for both `ROWS` and `RANGE` based window specifications. This change should allow the `LAST()` aggregation to run in multiple batches if the window is `[UNBOUNDED PRECEDING, CURRENT ROW`]. This will allow for much larger window and group sizes, because the group no longer needs to fit in GPU memory. This should help mitigate out-of-memory errors with `LAST()` window aggregations. Signed-off-by: MithunR <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Partially addresses #9299.
This commit adds support for batched execution of
FIRST()aggregations row-based window functions. This helps relax the erstwhile requirement that the window aggregation be executed over a single in-memory batch, thereby alleviating memory pressure and out-of-memory failures.