Separate out and clarify policies for providers #30657
Conversation
potiuk
requested review from
BasPH,
eladkal,
ashb,
pierrejeambrun,
jedcunningham,
dimberman,
ephraimbuddy,
bolkedebruin,
kaxil,
turbaszek,
mik-laj,
dstandish and
vikramkoka
potiuk
force-pushed
the
separate-out-and-clarify-approach-for-providers
branch
from
c8d4abd
to
efaa0bf
Compare
PROVIDERS.rst
Outdated
| Accepting new community providers should be a deliberate process that requires ``[DISCUSSION]`` | ||
| followed by ``[VOTE]`` thread at the airflow `devlist <https://airflow.apache.org/community/#mailing-list>`_. | ||
| In case of provider is integrating with an open-source software rather than service (particularly if the | ||
| open-source software is an Apache Software Foundation, Linux Software Foundation or similar organisation with | ||
| well established governance processes that are not strictly vendor-controlled) and when the software is | ||
| well established an popular it might be enough to have a good and complete PR of the provider, ideally | ||
| with a good test coverage (including integration tests) and documentation approved by the team | ||
| ``[LAZY CONSENSUS]`` on the devlist might be enough to accept the provider. |
There was a problem hiding this comment.
I think we have a loop hole here.
If I take for example the google case. The current state allows to add any new google integration without passing trough this procedure because everything is packed under 1 provider.
I think we should also specify handling case of adding new SDK under existed provider that extend the provider to a new service that wasn't part of it to begin with. If we take for example when we first added Tableau to the existed provider of Salesforce following the current policy states that there is no need for a mailing list thread which seems wrong?
There was a problem hiding this comment.
I am not sure we have to go that deep. I am more concerned about requests to add a new provider/integration thn a new "dependency" to an existing provider. We already get "historically" some providers that we manage, and that's sometihing that is both - the strength of Airflow and weakeness (dependency). And in general, any committer might block merging a new dependency being added when we think it is a problem. We do not have specifically explain that as a process (I think). Or when we see that a new operator does not fit. And we can always start a discussion at the devlist.
The process for adding a new provider is slightly different, I think - this is mostly about new integration, potentially new team of people from a service interacting with us, and a huge risk that they want to "drop" maintenance on us and won't be contributing new things, hoping that people in the community will and they can just walk away.
I think when getting new PR in an existing provider we have, we have much better knowledge - will there be maintenance, will there be support in case of problem, what happens when there is a need to upgrade or when the service gets disabled. We are going through that with Google team and while a bit late, the make a huge effort to make their provider updated now. This is a good lesson for us.
So I think while we could describe it in more detail, I tihnk this is really more of an explanation for anyone "new" (like Huawei Cloud service people) players that need to understand what it means to add a new provider.
Does it make sense?
There was a problem hiding this comment.
Also this is more a "default" behaviour. At any point, anyone can call for VOTE, LAZY CONSENSUS on just about anything, this is more like "what to expect" than "strict rules".
There was a problem hiding this comment.
I think when getting new PR in an existing provider we have, we have much better knowledge - will there be maintenance, will there be support in case of problem, what happens when there is a need to upgrade or when the service gets disabled. We are going through that with Google team and while a bit late, the make a huge effort to make their provider updated now. This is a good lesson for us.
I'm not referring to the case that we have now.
I'm referring for a case that adding a new extension for a service that is not part of the original provider scope. If tomorrow Google acquire a company technically it allows them to add a new integration for it without violating this policy. I don't mean to cases where under google.cloud we add more services and improve the support there i'm referring to cases of adding a whole new service branch like: leveldb, looker etc...
This change separates out the policies we have for providers to a separate PROVIERS.rst file. It also documents clearly the process and policy we have for accepting new community-managed providers, explaining the conditions that have to be fulfilled and stating a very strong preference of keeping providers maintained by the 3rd-party providers when there are 3rd-party teams that manage the providers.
potiuk
force-pushed
the
separate-out-and-clarify-approach-for-providers
branch
from
efaa0bf
to
d901c31
Compare
There was a problem hiding this comment.
Thank you for drafting this one!
I think we need also to clean up
https://airflow.apache.org/docs/apache-airflow-providers/index.html
Can I contribute my own provider to Apache Airflow?
this policy covers some of the content that we have in Airflow docs so maybe we just need to refer to this doc
Good point . Will take a look |
This change separates out the policies we have for providers to a separate PROVIERS.rst file. It also documents clearly the process and policy we have for accepting new community-managed providers, explaining the conditions that have to be fulfilled and stating a very strong preference of keeping providers maintained by the 3rd-party providers when there are 3rd-party teams that manage the providers.
We've recently clarified and described our policies for accepting providers to be maintained by the community (apache#30657) - this was directed towards the Airflow developers and contributors. This PR reviews user-facing part of the documentation for providers by removing some obsolete/not very useful documentation and pointing to the new policy where appropriate.
PR here #30816 |
We've recently clarified and described our policies for accepting providers to be maintained by the community (#30657) - this was directed towards the Airflow developers and contributors. This PR reviews user-facing part of the documentation for providers by removing some obsolete/not very useful documentation and pointing to the new policy where appropriate.
We've recently clarified and described our policies for accepting providers to be maintained by the community (#30657) - this was directed towards the Airflow developers and contributors. This PR reviews user-facing part of the documentation for providers by removing some obsolete/not very useful documentation and pointing to the new policy where appropriate. (cherry picked from commit 76ebc9b)
This change separates out the policies we have for providers to a separate PROVIERS.rst file. It also documents clearly the process and policy we have for accepting new community-managed providers, explaining the conditions that have to be fulfilled and stating a very strong preference of keeping providers maintained by the 3rd-party providers when there are 3rd-party teams that manage the providers.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.