Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix RollingGroupby and ExpandingGroupby to handle agg_columns. #1546

Merged
merged 3 commits into from
May 29, 2020

Conversation

ueshin
Copy link
Collaborator

@ueshin ueshin commented May 28, 2020

Fix Rolling and Expanding to handle agg_columns.

E.g.,

>>> kdf
   a    b
0  1  4.0
1  2  2.0
2  3  3.0
3  2  1.0
>>> kdf.groupby(kdf.a)[['b']].rolling(2).sum()
       a    b
a
1 0  NaN  NaN
3 2  NaN  NaN
2 1  NaN  NaN
  3  4.0  3.0

This should be:

>>> pdf.groupby(pdf.a)[['b']].rolling(2).sum()
       b
a
1 0  NaN
2 1  NaN
  3  3.0
3 2  NaN

Also, it raises errors for other cases:

>>> kdf.b.groupby(kdf.a).rolling(2).sum()
Traceback (most recent call last):
...
pyspark.sql.utils.AnalysisException: "cannot resolve '`a`' given input columns: ...

>>> kdf.groupby(kdf.a)['b'].rolling(2).sum()
Traceback (most recent call last):
...
pyspark.sql.utils.AnalysisException: "cannot resolve '`a`' given input columns: ...

@ueshin ueshin requested a review from HyukjinKwon May 28, 2020 00:01
@codecov-commenter
Copy link

codecov-commenter commented May 28, 2020

Codecov Report

Merging #1546 into master will increase coverage by 0.01%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1546      +/-   ##
==========================================
+ Coverage   94.18%   94.20%   +0.01%     
==========================================
  Files          38       38              
  Lines        8571     8571              
==========================================
+ Hits         8073     8074       +1     
+ Misses        498      497       -1     
Impacted Files Coverage Δ
databricks/koalas/groupby.py 90.31% <100.00%> (+0.01%) ⬆️
databricks/koalas/series.py 97.76% <100.00%> (+<0.01%) ⬆️
databricks/koalas/window.py 96.78% <100.00%> (+0.42%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e206de2...c6fdc76. Read the comment docs.

@ueshin ueshin changed the title Fix Rolling and Expanding to handle agg_columns. Fix RollingGroupby and ExpandingGroupby to handle agg_columns. May 28, 2020
Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I couldn't have time to take a close look but seems fine.

@ueshin
Copy link
Collaborator Author

ueshin commented May 29, 2020

Thanks! I'd merge this now.

@ueshin ueshin merged commit 35f3084 into databricks:master May 29, 2020
@ueshin ueshin deleted the window branch May 29, 2020 17:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants