Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade botocore/aiobotocore minimum requirements #33649

Merged
merged 1 commit into from
Aug 23, 2023

Conversation

potiuk
Copy link
Member

@potiuk potiuk commented Aug 23, 2023

Botocore has a very peculiar process of releasing new version every day, which means that it gives pip hard time to figure what will be the non-conflicting set of packages when we have too low of a minium version set as requirement.

Since we had > 1.24 that means that pip had to consider more than 340 versions for botocore, but also for related mypy packages and also a number of aiobotocore packages when resolving eager-upgrade.

We limit all the relevant packages to 1.28 as minimum version now, and we should continue doing that regularly in the future.


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@potiuk potiuk added the full tests needed We need to run full set of tests for this PR to merge label Aug 23, 2023
@potiuk potiuk closed this Aug 23, 2023
@potiuk potiuk reopened this Aug 23, 2023
@potiuk
Copy link
Member Author

potiuk commented Aug 23, 2023

Applied full tests needed to build for all Python versions. Let's see if the help given to pip by specifying higher versions of boto3-related packages will help.

Dockerfile.ci Show resolved Hide resolved
@potiuk potiuk added this to the Airflow 2.7.1 milestone Aug 23, 2023
@potiuk
Copy link
Member Author

potiuk commented Aug 23, 2023

ok. I think i will need to add a little "trick" for "pypi" providers to also help pip to resolve that (at least until we release newer version of the amazon provider with those limits included

Botocore has a very peculiar process of releasing new version
every day, which means that it gives `pip` hard time to figure
what will be the non-conflicting set of packages when we have
too low of a minium version set as requirement.

Since we had > 1.24 that means that `pip` had to consider
more than 340 versions for botocore, but also for related
mypy packages and also a number of aiobotocore packages when
resolving eager-upgrade.

We limit all the relevant packages to 1.28 as minimum version
now, and we should continue doing that regularly in the future.
@potiuk potiuk force-pushed the upgrade-botocore-requirements branch from 820af9c to b20bdb0 Compare August 23, 2023 12:03
@potiuk
Copy link
Member Author

potiuk commented Aug 23, 2023

All right - so fix for constraints was easy - the root cause was yandex we just resumed. I have to temporary disable it until we release it.

@potiuk potiuk closed this Aug 23, 2023
@potiuk potiuk reopened this Aug 23, 2023
@potiuk potiuk merged commit 5f504e9 into apache:main Aug 23, 2023
1 of 3 checks passed
@potiuk potiuk deleted the upgrade-botocore-requirements branch August 23, 2023 12:04
potiuk added a commit that referenced this pull request Aug 25, 2023
Botocore has a very peculiar process of releasing new version
every day, which means that it gives `pip` hard time to figure
what will be the non-conflicting set of packages when we have
too low of a minium version set as requirement.

Since we had > 1.24 that means that `pip` had to consider
more than 340 versions for botocore, but also for related
mypy packages and also a number of aiobotocore packages when
resolving eager-upgrade.

We limit all the relevant packages to 1.28 as minimum version
now, and we should continue doing that regularly in the future.

(cherry picked from commit 5f504e9)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:dev-tools area:providers full tests needed We need to run full set of tests for this PR to merge
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants