BUG: pyarrow dtype_backend incorrectly loads columns (from parquet) when the data stored is a list of structs and one of the struct fields has only None/null values

[kbuma]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

obj1 = {'a': [{'b': None}, 
              {'g': 'moo'}
              ]}

pd.DataFrame([obj1]).to_parquet("obj1.parq", engine="pyarrow")

a = pd.read_parquet("obj1.parq", engine="pyarrow")

print(a.a) 
print(a.a[0][0]) # success

a = pd.read_parquet("obj1.parq", engine="pyarrow", dtype_backend="pyarrow")

print(a.a) # note that the array is rendered as having one element instead of 2
print(a.a[0][0]) # ArrowIndexError

Issue Description

In this scenario the column data is reconstructed such that the underlying list object is invalid, ie somehow the list is created such that it contains all the structs as 1 element in the list rather than N while the list metadata thinks there are N elements.

It's unclear to me whether this is due to the implementation of pd.ArrowDType or the way it is called by the pyarrow Table to_pandas method when it is used as a types_mapper.

I've been able to only reproduce this issue when one of the struct fields only has None values defined and the dtype is inferred as null, e.g. in the example the column dtype is list<element: struct<b: null, g: string>>[pyarrow]

Expected Behavior

The expected behavior is shown in the example above when using the default dtype_backend.

Installed Versions

INSTALLED VERSIONS

commit : d9cdd2e
python : 3.11.8.final.0
python-bits : 64
OS : Darwin
OS-release : 23.4.0
Version : Darwin Kernel Version 23.4.0: Fri Mar 15 00:12:49 PDT 2024; root:xnu-10063.101.17~1/RELEASE_ARM64_T6020
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.UTF-8

pandas : 2.2.2
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0
setuptools : 69.2.0
pip : 24.0
Cython : 3.0.9
pytest : 8.1.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 3.1.3
IPython : 8.22.2
pandas_datareader : None
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.3.1
gcsfs : None
matplotlib : 3.8.3
numba : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 16.1.0
pyreadstat : None
python-calamine : None
pyxlsb : None
s3fs : 2024.3.1
scipy : 1.12.0
sqlalchemy : 2.0.28
tables : None
tabulate : 0.9.0
xarray : None
xlrd : None
zstandard : None
tzdata : 2024.1
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report - confirmed on main. It seems like we are in an invalid state here - a["a"] is a Series of length 1 but even a["a"].iloc[0] raises. Further investigations and PRs to fix are welcome!

[Angel-212]

take

[Angel-212]

The issue seems to stem from the upstream pyarrow package, specifically within the cast method of ChunkedArray.
The pd.read_parquet function reads the data from the file system
using pyarrow parquet.read_table. This returns a PyArrow table object.
The PyArrow table object (pa.Table) is converted to pandas via pa.Table.to_pandas()
passing a types_mapper argument of pandas.core.dtypes.ArrowDtype.
During the conversion to pandas within the to_pandas() call stack,
the ArrowDtype __from_arrow() method calls array.cast.
This seems to be creating an invalid ArrowExtensionArray which throws the
index exception when accessed.
The following snippet demonstrates the issue.

import pyarrow as pa
from pandas import ArrowDtype
from pandas.core.arrays.arrow.array import ArrowExtensionArray

fields = [
('b', pa.null()),
('g', pa.string()),
]

# Creating ArrowExtensionArray from ChunkedArray directly works
arr  = pa.array([[{'b': None, 'g': None}, {'b': None, 'g': 'moo'}]], type=pa.list_(pa.struct(fields)))
carr = pa.chunked_array(arr)
data = ArrowExtensionArray(carr)
print(data) # Works
print(data[0][0]) # Works

# ChunkedArray cast causes issue
pandas_dtype = ArrowDtype(arr.type)
ext_arr = carr.cast(pandas_dtype.pyarrow_dtype, safe=True)
print(ext_arr) # prints Invalid array but no exception
print(len(ext_arr)) # Len is 1
print(ext_arr[0]) # Raises ArrowIndexError

The snippet fails with:

line 21, in <module>
    print(ext_arr[0])
  File "pyarrow/scalar.pxi", line 124, in pyarrow.lib.Scalar.__str__
  File "pyarrow/scalar.pxi", line 704, in pyarrow.lib.ListScalar.as_py
  File "pyarrow/array.pxi", line 1604, in pyarrow.lib.Array.to_pylist
  File "pyarrow/array.pxi", line 1235, in __iter__
  File "pyarrow/array.pxi", line 1400, in pyarrow.lib.Array.getitem
  File "pyarrow/error.pxi", line 154, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
pyarrow.lib.ArrowIndexError: index with value of 1 is out-of-bounds for array of length 1

[rhshadrach]

Thanks @Angel-212 - stripping pandas from your reproducer:

fields = [('b', pa.null()),('g', pa.string())]
dtype = pa.list_(pa.struct(fields))
arr  = pa.array([[{'b': None, 'g': None}, {'b': None, 'g': 'moo'}]], type=dtype)
carr = pa.chunked_array(arr)
ext_arr = carr.cast(dtype, safe=True)
print(ext_arr) # prints Invalid array but no exception
print(len(ext_arr)) # Len is 1
print(ext_arr[0]) # Raises ArrowIndexError

Makes this a pyarrow issue. @Angel-212 - would you be able to open an issue with PyArrow?

[Angel-212]

Submitted apache/arrow#43838

[rhshadrach]

Going to leave this issue open to verify whether it gets fixed once the above issue is resolved.