BUG: Arrow backed series can't be displayed after string slicing it

[wirable23]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> k = pd.Series([f"AB{chr(127917)}C{chr(127917)}ㇱD"], dtype=pd.ArrowDtype(pa.string()))
>>> k
0    AB🎭C🎭ㇱD
dtype: string[pyarrow]
>>> k.str.slice(start=2,stop=None,step=4)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\series.py", line 1632, in __repr__
    return self.to_string(**repr_params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\series.py", line 1725, in to_string
    result = formatter.to_string()
             ^^^^^^^^^^^^^^^^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\io\formats\format.py", line 393, in to_string
    fmt_values = self._get_formatted_values()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\io\formats\format.py", line 377, in _get_formatted_values
    return format_array(
           ^^^^^^^^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\io\formats\format.py", line 1330, in format_array
    return fmt_obj.get_result()
           ^^^^^^^^^^^^^^^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\io\formats\format.py", line 1363, in get_result
    fmt_values = self._format_strings()
                 ^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\io\formats\format.py", line 1669, in _format_strings
    array = np.asarray(values)
            ^^^^^^^^^^^^^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\arrays\arrow\array.py", line 437, in __array__
    return self.to_numpy(dtype=dtype)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\ca2_ps_311_0515\Lib\site-packages\pandas\core\arrays\arrow\array.py", line 1076, in to_numpy
    result = np.asarray(self._data, dtype=dtype)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow\table.pxi", line 514, in pyarrow.lib.ChunkedArray.__array__
  File "pyarrow\table.pxi", line 495, in pyarrow.lib.ChunkedArray.to_numpy
  File "pyarrow\error.pxi", line 138, in pyarrow.lib.check_status
pyarrow.lib.ArrowException: Unknown error: Wrapping @���� failed
>>>

Issue Description

The series cannot be displayed after string slicing it. (likely invalid utf8 sequence remains).

Expected Behavior

I would expect a series that can be displayed

Installed Versions

INSTALLED VERSIONS

commit : 965ceca
python : 3.11.2.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.22621
machine : AMD64
processor : Intel64 Family 6 Model 85 Stepping 7, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 2.0.2
numpy : 1.24.3
pytz : 2023.3
dateutil : 2.8.2
setuptools : 67.7.2
pip : 23.1.2
Cython : None
pytest : 7.0.1
hypothesis : None
sphinx : 4.2.0
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.2
IPython : 8.13.2
pandas_datareader: None
bs4 : 4.12.2
bottleneck : None
brotli : None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 12.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : None
snappy : None
sqlalchemy : None
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
zstandard : None
tzdata : 2023.3
qtpy : None
pyqt5 : None

[lithomas1]

On main, I get "P". Is this what you expected?

>>> import pyarrow as pa
>>> k = pd.Series([f"AB{chr(127917)}C{chr(127917)}ㇱD"], dtype=pd.ArrowDtype(pa.string()))
>>> k
0    AB🎭C🎭ㇱD
dtype: string[pyarrow]
>>> k.str.slice(start=2, stop=None, step=4)
0    P
dtype: string[pyarrow]

[wirable23]

k = pd.Series([f"AB{chr(127917)}C{chr(127917)}ㇱD"], dtype=pd.StringDtype())
k.str.slice(start=2,stop=None,step=4)
0 🎭D
dtype: string

I was expecting the same behavior as the other StringDtype.

[lithomas1]

Thanks for the info.

I'll mark this as a bug, then. Haven't looked into whether its a bug in pandas or Arrow, though.

[wirable23]

Looks like a bug in Arrow:

pa.compute.utf8_slice_codeunits(f"AB{chr(127917)}C{chr(127917)}ㇱD", start=2, stop=None, step=4).as_py()

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pyarrow\scalar.pxi", line 632, in pyarrow.lib.StringScalar.as_py
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 4: invalid start byte
>>>

apache/arrow#36311

[mroeschke]

Looks like apache/arrow#36575 addressed this in pyarrow 13 so closing