[FEA] Support window operations on Decimal

[revans2]

Is your feature request related to a problem? Please describe.
This is still blocked on CUDF adding in support for decimal window operations. The window operations currently supported include:

• lead / lag
• count
• sum [FEA] Fixed-point precision support for SUM in a window operation rapidsai/cudf#7117
• row number
• max / min
• collect list

[mythrocks]

CUDF Decimal support in window functions:
min, max, count, lead, lag: rapidsai/cudf#7037
row_number: rapidsai/cudf#7061

[sperlingxx]

I have no idea what does "collect list" point to.

[sperlingxx]

I did some tests with cuDF java tests. I found that all window operations required by current issue can run on decimal columns except sum.
sum on decimal is still unsupported by cuDF because rolling with sum operation is yet supported on decimal: https://github.com/rapidsai/cudf/blob/branch-0.18/cpp/src/rolling/rolling_detail.hpp#L54

[revans2]

Thanks, then we should split this up into at least 2 issues. The first is for what we can support today. The second should be a follow on to implement sum, and then we can make sure that there is an issue in CUDF to track it. I asked @razajafri to take lead on making sure that all of the issues we need are in cudf so please coordinate on that.

The final issue is for collect_list which @mythrocks is working on a general cudf implementation for (we do not currently support it for any window operations).

[razajafri]

rapidsai/cudf#7061 and rapidsai/cudf#7037 are both merged in and can be supported by the plugin. I will create an issue for sum in cudf. As for collect_list @mythrocks is working on it. I spoke with him and he will create an issue in cudf to track that work

[razajafri]

I have opened a cudf issue to track sum in a window operation rapidsai/cudf#7117

[revans2]

Lead, Lag, Max, and Min were merged in as a part of #1512
Count was merged in as a part of #1476 but it was a mistake because it was intended to only go in for regular aggregate operations so tests for it were added as a part of #1512

Both sum and row_number are are merged into cudf so we should be able to get those merged in shortly.

[sperlingxx]

Hi @revans2, I met some problem on wrapping the sum support for decimal type. The problem is spark catalyst will conduct a hard-code precision promotion on the output decimal type of sum expression (code link), which leads to precision overflow any input decimal whose precision greater than 8.
Considering the output decimal type can pass to parent expressions recursively, I think perhaps we need create a catalyst rule to revert the precision promotion?

[revans2]

@sperlingxx We want to match what Spark does. We are just not going to be able to support sums greater than a precision of 8. This is the same limitation that we have for doing a sum with a group-by or a reduction in spark.

[sperlingxx]

Thanks @revans2. I created the pull request #1560 for sum. And I think the row_number is already supported for now?

[revans2]

@sperlingxx you are right row_number is done. I missed it because it really was a test only change.

[revans2]

@sameerz at this point we have everything done for this feature except collect_list. Do we want to keep this open for it, or should we split it off into a separate issue and close this?

[sameerz]

I am hoping we will see a collect_list PR in cudf soon, so let's keep this open. If it does not get raised in the next few days we can close this and open a new issue for collect_list.

[sameerz]

collect_list was merged in rapidsai/cudf#7189

[sameerz]

Follow up issue for supporting rank is in issue #1584 .