Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Match pandas scalar result types in reductions #9717

Merged

Conversation

brandon-b-miller
Copy link
Contributor

Moving this casting logic to python and updating it so that integer sum and product operations give back an int64 and give back the original column dtype in float cases. This is a breaking change.

Closes #8449

@brandon-b-miller brandon-b-miller added bug Something isn't working 2 - In Progress Currently a work in progress Python Affects Python cuDF API. breaking Breaking change labels Nov 17, 2021
@brandon-b-miller brandon-b-miller requested a review from a team as a code owner November 17, 2021 15:59
@brandon-b-miller brandon-b-miller self-assigned this Nov 17, 2021
@vyasr
Copy link
Contributor

vyasr commented Dec 3, 2021

@brandon-b-miller can you fix the style checks?

Comment on lines 293 to 294
def _resolve_reduction_dtype(self, reduction_op: str) -> Dtype:
return self.dtype
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we don't need this override?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we don't have this, we end up picking up the one from NumericalBaseColumn.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that's an indication that the definition in NumericalBaseColumn should be in NumericalColumn since that's the only case requiring special behavior. Then Decimal*Column will inherit the behavior of ColumnBase as expected.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah. duh

@@ -1239,6 +1239,13 @@ def _process_for_reduction(
)
return result_col

def _resolve_reduction_dtype(self, reduction_op: str) -> Dtype:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style nit: none of our methods follow a resolve_*() naming convention, so it feels kind of out of place. Perhaps just _reduction_result_dtype?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

@codecov
Copy link

codecov bot commented Dec 3, 2021

Codecov Report

Merging #9717 (2af1ca4) into branch-22.02 (967a333) will decrease coverage by 0.07%.
The diff coverage is n/a.

Impacted file tree graph

@@               Coverage Diff                @@
##           branch-22.02    #9717      +/-   ##
================================================
- Coverage         10.49%   10.41%   -0.08%     
================================================
  Files               119      119              
  Lines             20305    20480     +175     
================================================
+ Hits               2130     2134       +4     
- Misses            18175    18346     +171     
Impacted Files Coverage Δ
python/custreamz/custreamz/kafka.py 29.16% <0.00%> (-0.63%) ⬇️
python/dask_cudf/dask_cudf/sorting.py 92.30% <0.00%> (-0.61%) ⬇️
python/cudf/cudf/__init__.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/frame.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/index.py 0.00% <0.00%> (ø)
python/cudf/cudf/io/parquet.py 0.00% <0.00%> (ø)
python/cudf/cudf/core/series.py 0.00% <0.00%> (ø)
python/cudf/cudf/utils/utils.py 0.00% <0.00%> (ø)
python/cudf/cudf/utils/dtypes.py 0.00% <0.00%> (ø)
python/cudf/cudf/utils/ioutils.py 0.00% <0.00%> (ø)
... and 19 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 951f630...2af1ca4. Read the comment docs.

@brandon-b-miller brandon-b-miller added 3 - Ready for Review Ready for review by team and removed 2 - In Progress Currently a work in progress labels Dec 14, 2021
@brandon-b-miller
Copy link
Contributor Author

this should be ready for another look.

Copy link
Contributor

@vyasr vyasr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question to understand pandas behavior.

python/cudf/cudf/core/column/numerical.py Show resolved Hide resolved
@vyasr vyasr self-requested a review January 4, 2022 19:07
@brandon-b-miller
Copy link
Contributor Author

rerun tests

@brandon-b-miller
Copy link
Contributor Author

@gpucibot merge

@rapids-bot rapids-bot bot merged commit cc25f3d into rapidsai:branch-22.02 Jan 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 - Ready for Review Ready for review by team breaking Breaking change bug Something isn't working Python Affects Python cuDF API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Integer overflow in df.sum()
3 participants