Airflow Standard Provider #41564

romsharon98 · 2024-08-18T15:36:47Z

Extract all time operators and sensors from airflow core and remove them to standard provider
Mailing list thread: https://lists.apache.org/thread/2dmlqkcmyomm4q7rrovygs6bw655zx07

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

docs/apache-airflow-providers-core/index.rst

potiuk · 2024-08-25T11:28:30Z

One more thing - at least for now, "standard" should be added to list of preinstalled providers in hatch_build.py

PRE_INSTALLED_PROVIDERS = [
    "common.compat",
    "common.io",
    "common.sql",
    "fab>=1.0.2",
    "ftp",
    "http",
    "imap",
    "smtp",
    "sqlite",
]

potiuk · 2024-08-25T11:29:08Z

We might change where it is added in Airflow 3, but for now it should be added here.

eladkal

we are missing few things...

we need to deprecate the existed operators in core
we are missing all the operators and sensors we have in core (currently it's only for time operators)
I am not sure what about the hooks folder?

potiuk · 2024-08-25T11:34:12Z

we need to deprecate the existed operators in core

Yes, but I think this should be done as separate PR to v2-10-test. I think we do not want to back-port that one to v2-10-test - and we should simply do it there (after we release standard as provider). I am not sure we want to back-port the whole PR here to v2-10-test (providers are not build nor tested there anyway).

we are missing all the operators (currently it's only for time operators)

I think those should be separate, smaller PRs ?

I am not sure what about the hooks folder?

I think it's the same ^^ - some should be moved but separately.

gopidesupavan · 2024-08-25T11:36:31Z

Thanks @romsharon98 , will be adding rest of the operators and sensors to this.

gopidesupavan · 2024-08-25T11:37:50Z

we are missing few things...

we need to deprecate the existed operators in core

we are missing all the operators and sensors we have in core (currently it's only for time operators)

I am not sure what about the hooks folder?

@eladkal me and romsharon discussed , i will be adding rest of the operators and sensors.

potiuk · 2024-08-25T11:40:56Z

Ah .. so no ... We should not add it to preinstalled just yet - until we release it (sorry @romsharon98 -> see the failure on openapi client).

eladkal · 2024-08-25T11:43:26Z

Yes, but I think this should be done as separate PR to v2-10-test. I think we do not want to back-port that one to v2-10-test - and we should simply do it there (after we release standard as provider). I am not sure we want to back-port the whole PR here to v2-10-test (providers are not build nor tested there anyway).

Then I propose this PR to remove the files from main that we migrated to the provider.
The PR to v2-10-test would be just to replace the classes with deprecation notes after we release the provider.

potiuk · 2024-08-25T11:47:04Z

Then I propose this PR to remove the files from main that we migrated to the provider.

This is what the PR does.

eladkal · 2024-08-25T11:50:03Z

Ah! I didn't see changes to airflow/core folder so I assumed it was missed.
Not very friendly UX design from GitHub

gopidesupavan · 2024-08-25T11:50:56Z

Yes, but I think this should be done as separate PR to v2-10-test. I think we do not want to back-port that one to v2-10-test - and we should simply do it there (after we release standard as provider). I am not sure we want to back-port the whole PR here to v2-10-test (providers are not build nor tested there anyway).

Then I propose this PR to remove the files from main that we migrated to the provider. The PR to v2-10-test would be just to replace the classes with deprecation notes after we release the provider.

Please correct me if i understood this, on v2-10-test deprecation updates required for all the changes we are doing on main?

potiuk · 2024-08-25T11:54:21Z

Please correct me if i understood this, on v2-10-test deprecation updates required for all the changes we are doing on main?

No - only those that are supposed to affect DAG authors.

potiuk · 2024-08-25T12:02:01Z

BTW. We will have to make the old "standard" providers still work in Airflow 3 I think so we will have to make the old operator imports work (likely with the mechanism mentioned by @ashb - sys.meta_path) - otherwise it will require many dags to change. I think we should apply the same mechanism as we will do for importing BaseOperator etc. in Airflow 3 (and still raise deprecation warning).

@ashb ? Do you agree ?

potiuk · 2024-08-25T16:24:22Z

Interesting. The errors we see are very useful :).

examples are part of the documentation of "main" airflow" - and I am afraid they will have to remain as examples in "airflow" not moved to provider, especially because point 2)...
the examples should be embedded and shown in airflow when "examples" are enabled - I think we should not (at least not now) to develop a mechanism to read examples for Airflow from providers.
The tests should also mock the moved packages.

<module 'airflow.sensors.time_delta' from '/usr/local/lib/python3.8/site-packages/airflow/sensors/time_delta.py'> does not have the attribute 'sleep'
FAILED tests/providers/standard/time/sensors/test_time_delta.py::TestTimeDeltaSensorAsync::test_wait_sensor[True] - AttributeError: <module 'airflow.sensors.time_delta' from '/u

romsharon98 · 2024-08-25T20:02:43Z

Interesting. The errors we see are very useful :).

examples are part of the documentation of "main" airflow" - and I am afraid they will have to remain as examples in "airflow" not moved to provider, especially because point 2)...

the examples should be embedded and shown in airflow when "examples" are enabled - I think we should not (at least not now) to develop a mechanism to read examples for Airflow from providers.

The tests should also mock the moved packages.
<module 'airflow.sensors.time_delta' from '/usr/local/lib/python3.8/site-packages/airflow/sensors/time_delta.py'> does not have the attribute 'sleep'
FAILED tests/providers/standard/time/sensors/test_time_delta.py::TestTimeDeltaSensorAsync::test_wait_sensor[True] - AttributeError: <module 'airflow.sensors.time_delta' from '/u

revert the example dags to be under main airflow.
should I add a link in index.yaml for those example dags?

also not sure but should I change somewhere in the code to tell that this provider is mandatory and will auto installed?

potiuk · 2024-08-25T20:04:37Z

also not sure but should I change somewhere in the code to tell that this provider is mandatory and will auto installed?

That's the preinstalled_providers - but it will only work after it's been released and it will likely change with Airflow 3 so for now - no need.

…om python 2.8 and 2.9 competability

* add core.time provider * add source-date-epoch * change core to essentials * revert external task sensor location * add provider to airflow_providers_bug_report list * change new provider name to standard * change changelog * add integration * add standard provider to pre-install list * revert hatch_build * move examples back to airflow core * fixing mock * add title to commit * fix time.rst * change sensors example dags paths * revert example get_test_run addition * remove init * revert howto docs * fix docs * update standard provider airflow minimum version, remove standarad from python 2.8 and 2.9 competability * change provider as not-ready * add changelog * move how to docs to provider * fix docs images * fix import error * revert global constant * change time_sensors.py to time.py * revert min version to be 2.10 * remove providers from 2.9 and 2.8 * revert global constant * add operators guid to provider yaml

boring-cyborg bot added area:dev-tools area:providers labels Aug 18, 2024

nevcohen reviewed Aug 19, 2024

View reviewed changes

docs/apache-airflow-providers-core/index.rst Outdated Show resolved Hide resolved

romsharon98 force-pushed the feature/standard-provider branch from d179424 to f2f5fbd Compare August 19, 2024 16:26

romsharon98 changed the title ~~Airflow Core Provider~~ Airflow Standard Provider Aug 25, 2024

romsharon98 force-pushed the feature/standard-provider branch from daf5108 to 7b9296e Compare August 25, 2024 05:45

romsharon98 self-assigned this Aug 25, 2024

romsharon98 marked this pull request as ready for review August 25, 2024 11:16

romsharon98 requested review from potiuk, ashb and jedcunningham as code owners August 25, 2024 11:16

potiuk approved these changes Aug 25, 2024

View reviewed changes

eladkal reviewed Aug 25, 2024

View reviewed changes

romsharon98 force-pushed the feature/standard-provider branch from ee24a98 to aa7fd34 Compare August 25, 2024 11:35

romsharon98 force-pushed the feature/standard-provider branch from cad107f to 78b8d2a Compare August 25, 2024 13:39

romsharon98 added 12 commits September 17, 2024 09:23

fixing mock

5a52709

add title to commit

169134f

fix time.rst

44477b4

change sensors example dags paths

3bff8cd

revert example get_test_run addition

107afee

remove init

8cbd0c4

revert howto docs

12f3cdc

fix docs

de502cd

update standard provider airflow minimum version, remove standarad fr…

16ec49b

…om python 2.8 and 2.9 competability

change provider as not-ready

d6d5121

add changelog

99bc9fb

move how to docs to provider

27708ff

romsharon98 force-pushed the feature/standard-provider branch from 9760b46 to 27708ff Compare September 17, 2024 06:23

romsharon98 added 8 commits September 17, 2024 10:53

fix docs images

b4ad015

fix import error

1078d70

revert global constant

01555f4

change time_sensors.py to time.py

0632860

revert min version to be 2.10

fe2ff01

remove providers from 2.9 and 2.8

4b08a30

revert global constant

2d5d724

add operators guid to provider yaml

81f01c0

eladkal approved these changes Sep 18, 2024

View reviewed changes

romsharon98 merged commit 20ea6b7 into apache:main Sep 18, 2024
108 checks passed

gopidesupavan mentioned this pull request Oct 9, 2024

Remove deprecated param from BranchDayOfWeekOperator #41441

Closed

eladkal added the provider:standard label Oct 31, 2024

This was referenced Nov 2, 2024

Redirect old location module imports to standard provider #43610

Merged

Move core operators/sensors/triggers to standard provider #43641

Open

eladkal mentioned this pull request Nov 14, 2024

Status of testing Providers that were prepared on November 14, 2024 #44041

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Airflow Standard Provider #41564

Airflow Standard Provider #41564

romsharon98 commented Aug 18, 2024 •

edited

Loading

potiuk commented Aug 25, 2024

potiuk commented Aug 25, 2024

eladkal left a comment •

edited

Loading

potiuk commented Aug 25, 2024 •

edited

Loading

gopidesupavan commented Aug 25, 2024

gopidesupavan commented Aug 25, 2024 •

edited

Loading

potiuk commented Aug 25, 2024

eladkal commented Aug 25, 2024

potiuk commented Aug 25, 2024

eladkal commented Aug 25, 2024

gopidesupavan commented Aug 25, 2024

potiuk commented Aug 25, 2024

potiuk commented Aug 25, 2024

potiuk commented Aug 25, 2024

romsharon98 commented Aug 25, 2024

potiuk commented Aug 25, 2024

Airflow Standard Provider #41564

Airflow Standard Provider #41564

Conversation

romsharon98 commented Aug 18, 2024 • edited Loading

potiuk commented Aug 25, 2024

potiuk commented Aug 25, 2024

eladkal left a comment • edited Loading

Choose a reason for hiding this comment

potiuk commented Aug 25, 2024 • edited Loading

gopidesupavan commented Aug 25, 2024

gopidesupavan commented Aug 25, 2024 • edited Loading

potiuk commented Aug 25, 2024

eladkal commented Aug 25, 2024

potiuk commented Aug 25, 2024

eladkal commented Aug 25, 2024

gopidesupavan commented Aug 25, 2024

potiuk commented Aug 25, 2024

potiuk commented Aug 25, 2024

potiuk commented Aug 25, 2024

romsharon98 commented Aug 25, 2024

potiuk commented Aug 25, 2024

romsharon98 commented Aug 18, 2024 •

edited

Loading

eladkal left a comment •

edited

Loading

potiuk commented Aug 25, 2024 •

edited

Loading

gopidesupavan commented Aug 25, 2024 •

edited

Loading