Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Matching the behaviour to pandas>=1.0.0 at most. #1299

Merged
merged 11 commits into from
Feb 27, 2020

Conversation

itholic
Copy link
Contributor

@itholic itholic commented Feb 21, 2020

Follow-up for #1197

Since we're following latest version of pandas, should fix several TODOs with matching pandas>=1.0.0 for now.

For example.

the behaviour of Expanding.count() and ExpandingGroupby.count() are different depending on what pandas version has been installed.

  • pandas < 1.0.0
>>> s = pd.Series([2, 2, 3, 3, 3, 4, 4, 4, 4, 5, 5])
>>> s.groupby(s).expanding(3).count().sort_index()
2  0     1.0
   1     2.0
3  2     1.0
   3     2.0
   4     3.0
4  5     1.0
   6     2.0
   7     3.0
   8     4.0
5  9     1.0
   10    2.0
dtype: float64
  • pandas >= 1.0.0
>>> s = pd.Series([2, 2, 3, 3, 3, 4, 4, 4, 4, 5, 5])
>>> s.groupby(s).expanding(3).count().sort_index()
2  0     NaN
   1     NaN
3  2     NaN
   3     NaN
   4     3.0
4  5     NaN
   6     NaN
   7     3.0
   8     4.0
5  9     NaN
   10    NaN
dtype: float64

Since we're following latest version of pandas, need to fix this.

@codecov-io
Copy link

codecov-io commented Feb 21, 2020

Codecov Report

Merging #1299 into master will decrease coverage by 2.18%.
The diff coverage is 91.88%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1299      +/-   ##
==========================================
- Coverage   95.07%   92.89%   -2.19%     
==========================================
  Files          34       34              
  Lines        7230     7331     +101     
==========================================
- Hits         6874     6810      -64     
- Misses        356      521     +165
Impacted Files Coverage Δ
databricks/koalas/frame.py 93.54% <ø> (-2.98%) ⬇️
databricks/koalas/numpy_compat.py 90.9% <100%> (ø) ⬆️
databricks/koalas/missing/frame.py 100% <100%> (ø) ⬆️
databricks/koalas/missing/common.py 100% <100%> (ø) ⬆️
databricks/koalas/missing/__init__.py 100% <100%> (ø) ⬆️
databricks/koalas/mlflow.py 94.87% <100%> (ø) ⬆️
databricks/koalas/missing/groupby.py 100% <100%> (ø) ⬆️
databricks/koalas/indexes.py 96.13% <100%> (+0.22%) ⬆️
databricks/koalas/missing/window.py 100% <100%> (ø) ⬆️
databricks/koalas/missing/indexes.py 100% <100%> (ø) ⬆️
... and 40 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 038ae29...e09d040. Read the comment docs.

databricks/koalas/window.py Outdated Show resolved Hide resolved
@itholic itholic changed the title Following pandas1.0.0 for Expanding.count & ExpandingGroupby.count [WIP] Matching the behaviour to pandas>=1.0.0 at most. Feb 21, 2020
@HyukjinKwon
Copy link
Member

@itholic do you mind rebasing and syncing to the master? There are many conflicts as of 30b3334

@itholic
Copy link
Contributor Author

itholic commented Feb 22, 2020

@HyukjinKwon okay i did it :)

@itholic
Copy link
Contributor Author

itholic commented Feb 26, 2020

okay, i think the remained TODOs are almost solved, except #1299 (comment) which we decided to solve in separated PR.

See the #1299 (comment) for detail of Resolved TODOs.

@itholic itholic changed the title [WIP] Matching the behaviour to pandas>=1.0.0 at most. Matching the behaviour to pandas>=1.0.0 at most. Feb 26, 2020
Copy link
Member

@HyukjinKwon HyukjinKwon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if tests pass

@HyukjinKwon HyukjinKwon merged commit 84da886 into databricks:master Feb 27, 2020
@itholic itholic deleted the expanding_count_pandas1.0.0 branch September 10, 2020 11:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants