Avoid pyarrow.fs import for local storage

[rjzamora]

Description

This is not a resolution, but may help mitigate problems from aws/aws-sdk-cpp#2681

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[pentschev]

I'm +1000 on this change. Thanks @rjzamora !

[wence-]

Minor suggestions, but LGTM

[wence-]

Is this sufficient? The import of pyarrow.dataset (here https://github.com/rapidsai/cudf/blob/branch-23.12/python/cudf/cudf/io/parquet.py#L18) also seems to provoke importing pyarrow._fs.

Can we add a test (maybe run in a subprocess) that checks that import cudf; "pyarrow._fs" in sys.modules returns False?

[bdice]

I support @wence-'s request for a test but otherwise LGTM.

[rjzamora]

Is this sufficient? The import of pyarrow.dataset (here https://github.com/rapidsai/cudf/blob/branch-23.12/python/cudf/cudf/io/parquet.py#L18) also seems to provoke importing pyarrow._fs.

This is a good point. However, do we care about pyarrow._fs, or just pyarrow._s3fs? As far as I can tell, importing pyarrow.dataset does bring in pyarrow._fs, but pyarrow._s3fs isn't loaded unless pyarrow.fs is imported.

import sys
import pyarrow
pa_mods = set(sys.modules)
_fs = "pyarrow._fs"
_s3fs = "pyarrow._s3fs"

print("import pyarrow.dataset")
import pyarrow.dataset
ds_mods = set(sys.modules)
print(f"{_fs} imported: {_fs in (ds_mods - pa_mods)}")
print(f"{_s3fs} imported: {_s3fs in (ds_mods - pa_mods)}")

print("import pyarrow.fs")
import pyarrow.fs
fs_mods = set(sys.modules)
print(f"{_s3fs} imported: {_s3fs in (fs_mods - ds_mods)}")

import pyarrow.dataset
pyarrow._fs imported: True
pyarrow._s3fs imported: False
import pyarrow.fs
pyarrow._s3fs imported: True

Can we add a test (maybe run in a subprocess) that checks that import cudf; "pyarrow._fs" in sys.modules returns False?

A test is a good idea. Let me know if you think that we can just check that pyarrow._s3fs isn't imported.

[rjzamora]

I have some bad news. I started adding the test suggested by @wence- (thanks Lawrence!). However, I then discovered that pyarrow indeed imports pyarrow.fs/pyarrow._s3fs in other problematic places. For example:

In [1]: import sys
In [2]: mods = set(sys.modules)
In [3]: import pyarrow.parquet as pq
In [4]: after_mods = set(sys.modules)
In [5]: "pyarrow._s3fs" in after_mods
Out[5]: True

[rjzamora]

This test is useful, but adding it has rained a bit on our mini parade. We now see that the pyarrow._s3fs problem pops up whenever we import pyarrow.parquet or pyarrow.orc. (So, much less constrained than the "remote storage only" case we originally expected).

[pentschev]

Thanks for doing that investigation Rick. I see you've already adjusted the remainder of the code to take that into consideration and this test is still passing, so I'm good with things as they are now.

[pentschev]

Thanks for doing that investigation Rick. I see you've already adjusted the remainder of the code to take that into consideration and this test is still passing, so I'm good with things as they are now.

[pentschev]

Given sentiment was positive regarding this change and to reduce probability of segfaults in Dask et al., I'm merging this. Thanks a lot @rjzamora for working on this change!

[pentschev]

/merge