Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Python][Azure][Doc] Add documentation about AzureFilesystem #41496

Open
sugibuchi opened this issue May 2, 2024 · 1 comment
Open

[Python][Azure][Doc] Add documentation about AzureFilesystem #41496

sugibuchi opened this issue May 2, 2024 · 1 comment

Comments

@sugibuchi
Copy link

Describe the enhancement requested

#39968, which is included in version 16.0.0 release, added a Python binding of C++ AzureFileSystem to the PyArrow API.

However, (1) this addition of the native file system implementation for Azure has not yet been documented, and (2) this addition causes a backward compatibility issue in Pandas.

Documentation of the API and usage

We should document AzureFileSystem in the API reference and its usage in the user guide.

Note about the backward compatibility in Pandas

Pandas' read_parquet() and to_parquet() with abfs:// have stopped working in specific cases since PyArrow 16.0.0 due to the addition of AzureFileSystem in PyArrow.

Pandas implement a logic that first tries to get a PyArrow native file system implementation for a given URL and then falls back to fsspec if PyArrow does not have a native implementation for the URL.

https://github.com/pandas-dev/pandas/blob/v2.2.2/pandas/io/parquet.py#L116-L124

Due to this fallback logic, Pandas's read_parquet() and to_parquet() always use fssepc with PyArrow before 16.0.0.

With PyArrow 16.0.0, Pandas automatically uses PyArrow's native AzureFileSystem. However, this AzureFileSystem does not use authentication settings set in fsspec's global configuration. Instead, we must explicitly provide an authentication setting to read_parquet() and to_parquet() as storage_options independently of fsspec.

We need to figure out where and how we should document this backward compatibility issue.

Component(s)

Python

@jorisvandenbossche
Copy link
Member

We should definitely add the new AzureFileSystem to the pyarrow docs (both in the user guide https://arrow.apache.org/docs/dev/python/filesystems.html as reference guide https://arrow.apache.org/docs/dev/python/api/filesystems.html).

But for the pandas compatibility, I think it is best to explain this issue in the pandas docs somewhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants