Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compute Index.is_monotonic/Index.is_monotonic_decreasing in a distributed manner #1354

Merged

Conversation

HyukjinKwon
Copy link
Member

@HyukjinKwon HyukjinKwon commented Mar 18, 2020

This PR proposes to compute is_monotonic/is_monotonic_decreasing in a distributed manner.

The approach is:

  1. Check if it's ordered within each partition
  2. Calculate min/max from each partition and check if the min/max of partitions are ordered.

@HyukjinKwon HyukjinKwon force-pushed the distributed-monotonically-inc-dec branch 4 times, most recently from 7060e77 to 534148b Compare March 18, 2020 15:54
@HyukjinKwon
Copy link
Member Author

Basically, it checks:

  1. If the data in each partition is ordered
  2. If each partition's min/max are ordered.

@HyukjinKwon HyukjinKwon force-pushed the distributed-monotonically-inc-dec branch from 534148b to fb0b544 Compare March 19, 2020 06:37
@HyukjinKwon
Copy link
Member Author

Can't reproduce the test failure in my local ..

@HyukjinKwon HyukjinKwon force-pushed the distributed-monotonically-inc-dec branch from 30d9d37 to ae10be6 Compare March 19, 2020 10:26
@codecov-io
Copy link

codecov-io commented Mar 19, 2020

Codecov Report

Merging #1354 into master will increase coverage by 0.03%.
The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1354      +/-   ##
==========================================
+ Coverage   95.23%   95.27%   +0.03%     
==========================================
  Files          34       34              
  Lines        7576     7638      +62     
==========================================
+ Hits         7215     7277      +62     
  Misses        361      361              
Impacted Files Coverage Δ
databricks/koalas/indexes.py 97.00% <ø> (+0.24%) ⬆️
databricks/koalas/base.py 97.51% <100.00%> (+0.20%) ⬆️
databricks/koalas/indexing.py 94.55% <100.00%> (+0.02%) ⬆️
databricks/koalas/missing/frame.py 100.00% <0.00%> (ø)
databricks/koalas/missing/series.py 100.00% <0.00%> (ø)
databricks/koalas/missing/indexes.py 100.00% <0.00%> (ø)
databricks/koalas/frame.py 96.73% <0.00%> (+0.01%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update da3740d...1cda816. Read the comment docs.

@HyukjinKwon HyukjinKwon changed the title [WIP] Make is_monotonic/is_monotonic_decreasing distributed Make is_monotonic/is_monotonic_decreasing distributed for single Index Mar 19, 2020
@HyukjinKwon HyukjinKwon changed the title Make is_monotonic/is_monotonic_decreasing distributed for single Index Compute Index.is_monotonic/Index.is_monotonic_decreasing in a distributed manner Mar 19, 2020
@HyukjinKwon HyukjinKwon requested a review from ueshin March 19, 2020 13:22
Copy link
Collaborator

@ueshin ueshin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Otherwise, LGTM.

databricks/koalas/indexing.py Outdated Show resolved Hide resolved
@itholic
Copy link
Contributor

itholic commented Mar 19, 2020

Nice work

@itholic
Copy link
Contributor

itholic commented Mar 20, 2020

LGTM

@HyukjinKwon HyukjinKwon force-pushed the distributed-monotonically-inc-dec branch from cef272d to 1cda816 Compare March 20, 2020 02:49
@HyukjinKwon HyukjinKwon merged commit 663680b into databricks:master Mar 20, 2020
@HyukjinKwon HyukjinKwon deleted the distributed-monotonically-inc-dec branch September 11, 2020 07:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants