-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update description of installing providers separately from core #24454
Update description of installing providers separately from core #24454
Conversation
The description of how to install providers separately from the core was wrong. It suggested installing main constraints, but in fact we are not able to give any guarantees when installing providers this way so no constraints can guarantee consistent set of dependencies and the user is on their own in this case. This PR corrects the installation docs.
This is a result of talking to a user :). |
to install, upgrade or downgrade any of the providers you need. We release providers independently from the | ||
core of Airflow, so often new versions of providers are released before Airflow is, also if you do not want | ||
yet to upgrade Airflow to the latest version, you might want to install newly released providers separately. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
to install, upgrade or downgrade any of the providers you need. We release providers independently from the | |
core of Airflow, so often new versions of providers are released before Airflow is, also if you do not want | |
yet to upgrade Airflow to the latest version, you might want to install newly released providers separately. | |
to install, upgrade or downgrade any of the providers you need. We release providers independently from | |
core Airflow, so often new versions of providers are released before Airflow is. You also may might want to | |
install newly released providers separately without also upgrading your core Airflow version. |
What is the effort in creating: Currently we have constraint-2.3.2 which created one time only when version is released (so providers versions are never updated). Using constraint-main may cause problems as you deacribed in this PR. Basicly the idea/question/suggestion is to have a latest version of updated constraints per specific core version which will be updated regularly. This will save users the big headeach if doable. |
1st of all this is not for installing latest providers on "latest" airflow, but installing them on ANY compatible airflow. For this last scenario we are rather limited in what we can do. The overhead is not that big for just "latest" airflow - we are actually doing it every time when we release a bugfix release already - so 2.3.3 constraints are exactly that when we prepare them. And to be honest - it does not help a lot to anyone but those who are on the latest version of it is the "latest" only. Plus it is not always possible, we would have to likely manually remove some providers - And it's not really problematic actually. we have roughly 1.5- monthly release cycle for Airlfow bugfixes. So we are usually talking about max 2 weeks gap between those two events. And it will automatically use the latest providers, so I'd rather encourage people to update to latest bugfix release. Another possibility (hinted in the survey) - maybe even better we could also adapt our release schedule for providers - it does not have to be "exactly monthly" - even now it is "roughly monthly". So we could make sure we release all providers a week (or 1.5 week) before a new bugfix/feature release of Airlfow - making sure that when we release the "latest" it always has "just released" providers as well. That might actually be the easiest and best way to do. |
Should we merge it @eladkal ? |
The PR is likely ready to be merged. No tests are needed as no important environment files, nor python files were modified by it. However, committers might decide that full test matrix is needed and add the 'full tests needed' label. Then you should rebase it to the latest main or amend the last commit of the PR, and push it with --force-with-lease. |
The description of how to install providers separately from the core was wrong. It suggested installing main constraints, but in fact we are not able to give any guarantees when installing providers this way so no constraints can guarantee consistent set of dependencies and the user is on their own in this case. This PR corrects the installation docs. (cherry picked from commit 28d3236)
The description of how to install providers separately from the
core was wrong. It suggested installing main constraints, but
in fact we are not able to give any guarantees when installing
providers this way so no constraints can guarantee consistent
set of dependencies and the user is on their own in this case.
This PR corrects the installation docs.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragement file, named
{pr_number}.significant.rst
, in newsfragments.