Add optional features in providers. #21074
Conversation
potiuk
requested review from
mik-laj,
turbaszek and
vikramkoka
as code owners
boring-cyborg
bot
added
area:providers
kind:documentation
provider:google
potiuk
force-pushed
the
unhide-provider-import-errors
branch
3 times, most recently
from
0362f44 to
db48779
Compare
potiuk
force-pushed
the
unhide-provider-import-errors
branch
3 times, most recently
from
f2a254f to
19f2fdf
Compare
Some features in providers can be optional, depending on the presence of some libraries. Since Providers Manager tries to import the right classes that are exposed via providers it should not - in this case - log warning message for those optional features. Previously, all ImportErrors were turned into debug log but now we only turn them in debug log when creator of the provider deliberately raised an AirflowOptionalProviderFeatureException. Instructions on how to raise such exception in the way to keep backwards compatibility were updated in proider's documentation. Fixes: apache#20709
potiuk
force-pushed
the
unhide-provider-import-errors
branch
from
19f2fdf to
cddad92
Compare
That’s what |
github-actions
bot
added
the
full tests needed
|
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
Oh yeah.... Unfortunately in our case we have N independent gunicorns processes running and restarting every X seconds when you have kinda busy use of airflow and "restart every x requests" feature of gunicorn enabled (which most people do) So short of having a separate service for that or storing this in a DB I think we are out of luck :( BTW. I actually toyed with an idea to add UI notification for that BTW. Maybe that is actually a good idea (but this is the kind of notification that I think we have no support for yet - dismissable (and staying dismissed if user did dismiss them). Or am I wrong? |
Some features in providers can be optional, depending on the
presence of some libraries. Since Providers Manager tries
to import the right classes that are exposed via providers it
should not - in this case - log warning message for those
optional features. Previously, all ImportErrors were turned into
debug log but now we only turn them in debug log when creator
of the provider deliberately raised
an AirflowOptionalProviderFeatureException.
Instructions on how to raise such exception in the way to keep
backwards compatibility were updated in proider's documentation.
Fixes: #20709
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.