ENH: Proposed min s3fs 2021.5.0 is incompatible with moto. Should instead maintain s3fs>0.4.0 recommendation

[JMBurley]

The upcoming 1.5.0 pandas update proposes a minimum version of s3fs 2021.5.0 (see doc/source/whatsnew/v1.5.0.rst).

This will not be acceptable to any project that interacts with s3 professionally, due to s3fs extremely tight pinning to outdated botocore.

Specifically, s3fs>=0.5.0 requires aiobotocore which is very tightly binned to botocore, and that pinning is mutually incompatible with moto (some unresolved issues explaining the aiobotocore botocore problem). Therefore you cannot use s3fs and test s3 access in the same project without running into incompatible versions.

Attempting pip-compile on a project with

s3fs[boto3]>=2021.05.0  # what pandas is about to enforce
moto[s3]>=1.0  # incredibly lenient moto request

returns

  botocore<1.24.22,>=1.24.21 (from aiobotocore==2.3.4->s3fs[boto3]==2022.5.0->my-project (setup.py))
  botocore>=1.12.201 (from moto==3.1.16->my-project (setup.py))
  botocore<1.28.0,>=1.27.30 (from boto3==1.24.30->moto==3.1.16->my-project (setup.py))

even with latest pip-tools update to handling dependencies, moto-s3fs is still unresolvable.

I would strongly prefer pandas to not (even in optional dependencies) spread this problem further.

PS. It isn't actively documented, but right now the pandas testing is implicitly pinned to two s3fs releases from March 2020 because pandas needs moto for local tests and moto cannot coexist with s3fs>0.5.0 and pandas requires s3fs>0.4.0

Describe the solution you'd like

Do not update s3fs min requirement until packaging problems from upstream aiobotocore requirements are resolved.

[MarcoGorelli]

Thanks for the report - could you find the PR which set this minimum s3fs version please? Just to understand the context of the decision

[MarcoGorelli]

latest CI shows both the latest moto and s3fs 2021.11.0 , so I think this has been solved now?

closing for now then, please please do ping / reopen if I've misunderstood