[BUG] Integer overflow in df.sum()

[esnvidia]

Integer overflow occurs when calculating a sum over a dataframe column, but not in cupy

Steps/Code to reproduce bug

cupy.random.seed(314)
cupy.random.randint(2, size=10000, dtype=cupy.int8).sum()

Out: 5058

cupy.random.seed(314)
cudf.DataFrame(cupy.random.randint(2, size=10000, dtype=cupy.int8)).sum()
Out: 0 -62
dtype: int8

Expected behavior
Out: 5058

Environment overview (please complete the following information)
conda install of rapids-0.19

[shwina]

This is because, unlike CuPy, cuDF does not always assume int64 as the result type. Although maybe we should -- as this can lead to some hard-to-find bugs.

As a workaround, you could specify the result type:

In [32]: cupy.random.seed(314)
    ...: cudf.DataFrame(cupy.random.randint(2, size=10000, dtype=cupy.int8)).sum(dtype="int64")
Out[32]:
0    5058
dtype: int64

[brandon-b-miller]

Pandas yields an int64 it seems to matter what the dtype of the input column is:

>>> x = pd.Series([0], dtype='uint8')
>>> x.sum()
0
>>> type(x.sum())
<class 'numpy.int64'>

However it seems to have no actual guards against overflow either

>>> x = pd.Series([np.iinfo('int64').max, np.iinfo('int64').max])
>>> x
0    9223372036854775807
1    9223372036854775807
dtype: int64
>>> x.sum()
-2

IMO what we do about this depends on if we'd have to internally make an upcasted copy of the data to do the reduction over, vs if libcudf's output type provides at workaround for this. heres the header. From this it looks like it's clever enough to just accumulate the result into an int64 or something else if requested but this should be tested in case my reading of it isn't right.

[shwina]

We can expect libcudf to do the right thing here. From Python, we should specify int64 as the result type, depending on the reduction.

[brandon-b-miller]

That seems sensible to me, just wanted to make sure that wasn't translating to copy->cast->reduce, which it looks like it isn't.

[esnvidia]

Sounds good, it seems like it would also give issues with floats as well.

Related issue, but in cupy:

cupy.asarray(100*cupy.random.rand(10000), dtype=cupy.float16).sum()

Out: array(inf, dtype=float16)

cudf.DataFrame(cupy.asarray(100*cupy.random.rand(10000), dtype=cupy.float16)).sum()

Out: 0 494430.96875
dtype: float32

So sum of floats seem to cast to float32, Making the sequence larger I get inf again

cudf.DataFrame(cupy.asarray(100000*cupy.random.rand(100000000), dtype=cupy.float16)).sum()

0 inf
dtype: float32

Although this works when specifying sum dtype, which is very odd:

cudf.DataFrame(cupy.asarray(100000*cupy.random.rand(100000000), dtype=cupy.float32)).sum(dtype=cupy.float32)

0 4.999604e+12
dtype: float32

[github-actions]

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.