Add first-class dtype utilities

[vyasr]

This PR adds a new cudf.api.types module that aims to match pandas.api.types while providing the necessary compatibility layers for cudf objects that is missing from the corresponding pandas APIs. It also replaces most internal uses of pandas.api.types in an attempt to centralize all typing logic so that we have a single place in which to perform any special dtype handling as needed. This work is intended as a best-effort, first-pass attempt to isolate our dependence on pandas dtype APIs; while it resolves a number of incompatibilities with pandas and other unexpected behaviors, there are still a number of open questions that still need to be addressed to completely wrap this up. I've noted a number of TODOs in the code (for instance relating to our nested types or the different types of time types e.g. timedelta). Since getting all of these correct in a single PR will be almost impossible given the pervasive use of dtype utilities throughout our code, this PR is a good first step in that direction that I think we can merge and then work on incrementally fixing the outstanding issues rather than trying to perfect all our type handling here.

[shwina]

cudf.utils.dtypes.is_numerical_dtype has been moved to cudf.api.types.is_numeric_dtype to match pandas. I have also changed this function to return True for decimal types because that's what I would expect as a user. However, this change breaks some internal code that relies on decimals not being classified as numeric. We also have an is_decimal_dtype function. Are people OK with replacing internal usage (either with explicitly checking both functions or defining a third convenience utility is_non_decimal_numeric_dtype), or is there a good reason to leave the behavior as is?

• Personally, I'm fine with is_numeric_dtype() returning True for decimal types. It's a bit of a mental readjustment for me, but it makes sense from and end-user perspective.

• Internally we can do something like is_integer_dtype() or is_float_dtype(), or maybe for efficiency, is_integer_or_float_dtype()?

[...] introspecting the data. Should we do that in this case?

I don't think there's a way to do this without doing a pass over the entire data, which I'd be -1 for. Sadly, I think the best thing for us to do is return True for Series of arbitrary objects and let things fail downstream in such cases.

[brandon-b-miller]

👍 on is_numeric_dtype returning true for decimals. As for how to handle the situations internally that break from that change, since is_decimal_dtype excludes numeric non decimal, I imagine we can just check the dtype using that function and do something specific based on if it's true.

For is_scalar, I think it could be a source of bugs if it doesn't return the same thing as the pandas version certainly for pandas a numpy objects. If we rely on it doing that internally, I'd say we should change that. In my mind this function returns the same thing as pandas for everything the pandas function can handle, plus return true for cuDF scalars or other cuDF objects that we need it to. Not sure about 0d cupy arrays.

RE: the string dtype, I agree with @shwina that scanning should be avoided. Maybe since it lives on the host checking the 0'th element wouldn't be terrible? Although I guess that doesn't guarantee they're all strings. I wonder what the plan is for that on the pandas roadmap, if there are any plans? Since they have a true pd.StringDtype now it could be that this api changes in the future.

[vyasr]

Yeah I think we'd have to check the entire Series to be sure that it's all strings, not like some strings and some other stuff. That being the case, we may be stuck with letting that fail downstream for now.

Note to self, this PR will need to account for changes in #8332 that start introducing the cudf.api.types module for one specific case.

[codecov]

Codecov Report

❗ No coverage uploaded for pull request base (branch-21.08@2606b71). Click here to learn what that means.
The diff coverage is n/a.

❗ Current head 8762315 differs from pull request most recent head e61765a. Consider uploading reports for the commit e61765a to get more accurate results

@@               Coverage Diff               @@
##             branch-21.08    #8308   +/-   ##
===============================================
  Coverage                ?   82.59%           
===============================================
  Files                   ?      109           
  Lines                   ?    17865           
  Branches                ?        0           
===============================================
  Hits                    ?    14755           
  Misses                  ?     3110           
  Partials                ?        0           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2606b71...e61765a. Read the comment docs.

[shwina]

Looks really good to me. Fantastic work, @vyasr!

[vyasr]

@gpucibot merge

[vyasr]

rerun tests

[galipremsagar]

rerun tests