ERR/API: non-string types

[michaelaye]

xref #5602, #9343, #13806 (comments)

Code Sample, a copy-pastable example if possible

In [2]: s = pd.Series(['a', 123, 'b'])

In [3]: s.str.startswith('a')
Out[3]:
0     True
1      NaN
2    False
dtype: object

Expected Output

raise an exception
Reasoning: If I provide NaN, it's okay to return NaN, but silently convert is hard to find. The user cannot easily drop hidden integers in an 'O' series.

output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.2.final.0
python-bits: 64
OS: Darwin
OS-release: 14.5.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.18.1
nose: None
pip: 8.1.2
setuptools: 23.0.0
Cython: 0.24.1
numpy: 1.11.1
scipy: 0.18.0
statsmodels: None
xarray: 0.7.2
IPython: 5.0.0
sphinx: 1.4.1
patsy: None
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.6.1
matplotlib: 1.5.1
openpyxl: None
xlrd: 1.0.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999999999
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: None
pandas_datareader: None

[michaelaye]

Potential connection: #9343

[shoyer]

@michaelaye I fixed your example -- Series.startswith does not exist (you need to use the str prefix)

[michaelaye]

ah, sorry. thanks!

[sinhrks]

I think all related string methods should have following kwds to cover most of usecases:

• errors: whether to raise or coerce non-str input
• na: specify value to fill NaN (only exists in few methods)

pd.Series(['a', 2, 3]).str.startswith('a')
# 0    True
# 1     NaN
# 2     NaN
# dtype: object

pd.Series(['a', 2, 3]).str.startswith('a', na=False)
# 0     True
# 1    False
# 2    False
# dtype: bool

related to #5672 and #13806 (comments)

[TomAugspurger]

Part of this would be solved by a proper string type, since pd.Series(['a', 123, 'b']).str.* would raise an error. But given the type system we have today, a keyword seems reasonable.

na: specify value to fill NaN (only exists in few methods

Would fill_value be a better name? More consistent with other operations.

[jreback]

it is trivial to determine string types

This simply needs impl. also will deal with #9343

In [1]: pd.lib.infer_dtype(pd.Series(['a', 123, 'b']).values)
Out[1]: 'mixed-integer'

In [2]: pd.lib.infer_dtype(pd.Series(['a', np.nan, 'b']).values)
Out[2]: 'mixed'

In [3]: pd.lib.infer_dtype(pd.Series(['a', 'c', 'b']).values)
Out[3]: 'string'

There is a slight perf penalty for this, but prob worth it.

[jreback]

I like @sinhrks proposal, which unifies the API (and these same kwds are common in pandas codebase), e.g. errors=raise|ignore|coerce are pretty intuitve to select behavior. Note that we generally have errors='raise', thinking of datetimes, though here for boolean methods I think errors='coerce' is appropriate (e.g. a non-string that is by defintiion False).

[h-vetinari]

@sinhrks @TomAugspurger

Would fill_value be a better name? More consistent with other operations.

There are several methods with 'fill_value' (often also in combination with 'errors') - quick scan of doc/build/html/generated yields 72 hits. On the other hand, the .str.cat-method uses the name 'na_rep', although in a (very) slightly different way.