-
Notifications
You must be signed in to change notification settings - Fork 902
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] grouped rolling produces wrong answer for structs #8887
Comments
I ran into a related failure, while working on Since the |
That is great. It would be good to also understand if there is type checking that needs to be added in somewhere? Is this a problem for list columns too? |
Yes, I should think so.
Yes, this appears to be borked on |
I have verified that #9194 fixes this. To be safe, we should probably add a test for grouping on |
Fixes #8887. #9024 added support for `STRUCT` groupby keys. This commit adds a test for `grouped_rolling_window()` where the groupby keys are `STRUCT`. Authors: - MithunR (https://github.com/mythrocks) Approvers: - Vukasin Milovanovic (https://github.com/vuule) - David Wendt (https://github.com/davidwendt) URL: #9228
Describe the bug
The grouped rolling API lets structs in as a group by key, but produces incorrect answers.
Steps/Code to reproduce bug
This is almost exactly the same as the test above it, but
grp_col
is a struct column wrapping the originalgrp_col
. It now fails with an incorrect result.Expected behavior
I would love it if it could produce a correct result, but I am willing to put up with it throwing an exception for group by keys it cannot support.
Environment overview (please complete the following information)
I tested this on both 20.08 and 20.10. I think think it is a blocker in any way for us. I can disable it in the Spark plugin.
The text was updated successfully, but these errors were encountered: