Dependencies are specified with upper bounds, causing problems with resolution

[flying-sheep]

Please unpin your dependencies. Python libraries specifying upper version boundaries or even versions pinned using ~=x.y.z or == cause problems when combined with other libraries: https://iscinumpy.dev/post/bound-version-constraints/

Pinning dependencies is OK if you are a) hacky shim around another library that needs to reach deeply into its guts (like aiobotocore is to botocore) and also b) are able to very quickly react to new releases of your parent library. If either of the two doesn’t apply, don’t specify upper bounds.

That you do manifests in bugs like, #357, #558, and downstream issues that not always reach you like this, this, or one we recently ran into, where botocore and boto3 versions were rendered incompatible due to s3fs.

You can configure your CI to regularly run with newer versions of your dependencies to catch problems with updated dependencies early.

[martindurant]

What do you suggest that we do, when it is likely that newer releases of aiobotocore will break older versions of s3fs? The pin with fsspec is essential, they are always released together.

[flying-sheep]

The pin with fsspec is essential, they are always released together.

That’s fine then, you maintain both and if you choose to do that and stick to it, there’s no problem.

What do you suggest that we do, when it is likely that newer releases of aiobotocore will break older versions of s3fs?

Aiobotocore being tied to botocore’s version makes sense. Aiobotocore has to hack deeply into botocore’s internals since it tries to make it do things it wasn’t designed for. That’s a brave endeavor, and limiting breakage by tying versions together here makes sense. It needs very active maintenance to work out though.

S3fs relies on aiobotocore’s abstractions to provide specific functionality. It should be able to only rely on stable APIs in aiobotocore. If it can’t do that, aiobotocore should have been discarded in the evaluation phase. If it can’t be relied on to provide stable APIs, it’s alpha quality software and shouldn’t be used as a dependency yet.

Luckily, 7ff31c5 implies it has a stable API. You just bumped the version without doing anything else. If you’d depend on aiobotocore>=2.1.0 instead, you wouldn’t have had to act at all and #558 would have never been filed.

[martindurant]

I can't really comment on whether aiobotocore is "alpha" or "beta", but we absolutely need it, as nothing else does what it does. We have indeed found cases where things have changed and broken our usage, but probably somewhat deeper than a typical user. See the code translating a/boto exceptions.

Luckily, 7ff31c5 implies it has a stable API. You just bumped the version without doing anything else. If you’d depend on aiobotocore>=2.1.0 instead, you wouldn’t have had to act at all and #558 would have never been filed.

We didn't know it would be problem free in this case. Previous version changes were not without problems, and once the package is on PyPI, we cannot change its deps to indicate that it is incompatible with some later version of aiobotocore. We choose to be conservative, and update our pin as soon as we can, and have a rapid release cycle ourselves.

[flying-sheep]

we absolutely need

I think this, combined with the hackiness of botocore’s implementation is what leads to problems here.

Don’t get me wrong, I love Python’s async and there’s other projects that have the same problems (e.g. numba doing its byte code patching), but until aiobotocore is the upstream and botocore wraps it (which could be done non-hackily), there will be problems for anyone depending on it.

[martindurant]

While you may well be right, I don't see that there's anything we can do about it in this repo. Indeed, I would love to see aio supported in botocore itself (I am in an issue to this effect on their tracker). Do you have any suggestions?

[flying-sheep]

Since you said you positively need aiobotocore, I have nothing productive at this point.

I think an alternative for them would be to do their own codegen instead of error-prone monkeypatching. Or get talking with botocore people until they agree to collaborate & merge a PR fixing boto/botocore#458