API: is_string_dtype is not strict

[jreback]

#15533 (comment)

pandas.types.common.is_string_dtype is not strict, just checking if its a fixed-width numpy string/unicode type or object dtype. It could be made strict via something like this.

def is_string_dtype(arr_or_dtype):
    dtype = _get_dtype(arr_or_dtype)
    if isinstance(arr_or_dtype, np.ndarray):
        if not lib.infer_dtype(arr_or_dtype) in ['string', 'unicode']:
            return False

    return dtype.kind in ('O', 'S', 'U') and not is_period_dtype(dtype)

this would need performance checking to see if its a problem. Further this then changes the API a tiny bit (as we allow an object OR a dtype to be passed in). Which is probably ok as long as its documented a bit.

[jreback]

xref @ResidentMario

[ResidentMario]

There definitely needs to be a strict implementation somewhere in types, I have a working-ish solution for #15533 that depends on this.

Maybe also consider leaving this method as-is and throwing in a new is_string_dtype_strict method with this signature, if performance is a problem.

How to test performance? Will timeit do, or need to add something to asv_bench?

[jreback]

asv is the place for things like this.

[jreback]

@ResidentMario you shouldn't need this fyi

[ivirshup]

Any updates on this? It seems like behavior changed in v0.24.0, but not necessarily for the better.

import pandas as pd
from pandas.api.types import is_string_dtype

is_string_dtype(pd.Series(pd.Categorical([1,2,3])))
# False in v0.23.4
# True in v0.24.1

# Still:
is_string_dtype(pd.Series([(0,1), (1,1)]))
# True in v0.23.4
# True in v0.24.1

This was causing issues in a project where we were expecting the results of this check not to change between versions. On further investigation, it seems like this function is unreliable (tuples aren't strings, categoricals as strings feels very R).