Skip to content

Commit

Permalink
Schedule plexus and apache.sqoop providers for removal (#36208)
Browse files Browse the repository at this point in the history
The providers have been agreed to be removed in the LAZY CONSENSUS
threads:

* https://lists.apache.org/thread/6so1qh8kf088spd2y4nl0o5tpbyn0x3b
* https://lists.apache.org/thread/wnov3n2m2v0blnxl3q413mj8vphdrz4b

(cherry picked from commit 97e8f58)
  • Loading branch information
potiuk committed Dec 15, 2023
1 parent 108b305 commit 5422ceb
Show file tree
Hide file tree
Showing 17 changed files with 62 additions and 78 deletions.
2 changes: 0 additions & 2 deletions .github/ISSUE_TEMPLATE/airflow_providers_bug_report.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,6 @@ body:
- apache-pig
- apache-pinot
- apache-spark
- apache-sqoop
- apprise
- arangodb
- asana
Expand Down Expand Up @@ -88,7 +87,6 @@ body:
- papermill
- pgvector
- pinecone
- plexus
- postgres
- presto
- redis
Expand Down
14 changes: 7 additions & 7 deletions CONTRIBUTING.rst
Original file line number Diff line number Diff line change
Expand Up @@ -670,18 +670,18 @@ This is the full list of those extras:
.. START EXTRAS HERE
aiobotocore, airbyte, alibaba, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra,
apache.drill, apache.druid, apache.flink, apache.hdfs, apache.hive, apache.impala, apache.kafka,
apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark, apache.sqoop, apache.webhdfs,
apprise, arangodb, asana, async, atlas, atlassian.jira, aws, azure, cassandra, celery, cgroups,
cloudant, cncf.kubernetes, cohere, common.io, common.sql, crypto, databricks, datadog, dbt.cloud,
apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark, apache.webhdfs, apprise,
arangodb, asana, async, atlas, atlassian.jira, aws, azure, cassandra, celery, cgroups, cloudant,
cncf.kubernetes, cohere, common.io, common.sql, crypto, databricks, datadog, dbt.cloud,
deprecated_api, devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc, doc_gen, docker,
druid, elasticsearch, exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google,
google_auth, grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, kerberos, kubernetes,
ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo, mssql,
mysql, neo4j, odbc, openai, openfaas, openlineage, opensearch, opsgenie, oracle, otel, pagerduty,
pandas, papermill, password, pgvector, pinecone, pinot, plexus, postgres, presto, rabbitmq, redis,
s3, s3fs, salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp,
snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv,
weaviate, webhdfs, winrm, yandex, zendesk
pandas, papermill, password, pgvector, pinecone, pinot, postgres, presto, rabbitmq, redis, s3, s3fs,
salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp, snowflake,
spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv, weaviate,
webhdfs, winrm, yandex, zendesk
.. END EXTRAS HERE
Provider packages
Expand Down
14 changes: 7 additions & 7 deletions INSTALL
Original file line number Diff line number Diff line change
Expand Up @@ -96,18 +96,18 @@ The list of available extras:
# START EXTRAS HERE
aiobotocore, airbyte, alibaba, all, all_dbs, amazon, apache.atlas, apache.beam, apache.cassandra,
apache.drill, apache.druid, apache.flink, apache.hdfs, apache.hive, apache.impala, apache.kafka,
apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark, apache.sqoop, apache.webhdfs,
apprise, arangodb, asana, async, atlas, atlassian.jira, aws, azure, cassandra, celery, cgroups,
cloudant, cncf.kubernetes, cohere, common.io, common.sql, crypto, databricks, datadog, dbt.cloud,
apache.kylin, apache.livy, apache.pig, apache.pinot, apache.spark, apache.webhdfs, apprise,
arangodb, asana, async, atlas, atlassian.jira, aws, azure, cassandra, celery, cgroups, cloudant,
cncf.kubernetes, cohere, common.io, common.sql, crypto, databricks, datadog, dbt.cloud,
deprecated_api, devel, devel_all, devel_ci, devel_hadoop, dingding, discord, doc, doc_gen, docker,
druid, elasticsearch, exasol, facebook, ftp, gcp, gcp_api, github, github_enterprise, google,
google_auth, grpc, hashicorp, hdfs, hive, http, imap, influxdb, jdbc, jenkins, kerberos, kubernetes,
ldap, leveldb, microsoft.azure, microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo, mssql,
mysql, neo4j, odbc, openai, openfaas, openlineage, opensearch, opsgenie, oracle, otel, pagerduty,
pandas, papermill, password, pgvector, pinecone, pinot, plexus, postgres, presto, rabbitmq, redis,
s3, s3fs, salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp,
snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv,
weaviate, webhdfs, winrm, yandex, zendesk
pandas, papermill, password, pgvector, pinecone, pinot, postgres, presto, rabbitmq, redis, s3, s3fs,
salesforce, samba, saml, segment, sendgrid, sentry, sftp, singularity, slack, smtp, snowflake,
spark, sqlite, ssh, statsd, tableau, tabular, telegram, trino, vertica, virtualenv, weaviate,
webhdfs, winrm, yandex, zendesk
# END EXTRAS HERE

# For installing Airflow in development environments - see CONTRIBUTING.rst
Expand Down
11 changes: 6 additions & 5 deletions airflow/providers/MANAGING_PROVIDERS_LIFECYCLE.rst
Original file line number Diff line number Diff line change
Expand Up @@ -514,14 +514,15 @@ providers (and then remove all the code and documentation related to the provide

The "removed: true" flag will cause the provider to be available for the following commands (note that such
provider has to be explicitly added as selected to the package - such provider will not be included in
the available list of providers):
the available list of providers or when documentation is built unless --include-removed-providers
flag is used):

* ``breeze build-docs``
* ``breeze release-management prepare-provider-documentation``
* ``breeze release-management prepare-provider-packages``
* ``breeze release-management publish-docs``

For all those commands, release manager needs to specify such to-be-removed provider explicitly as extra
command during the release process. Except the changelog that needs to be maintained manually, all other
documentation (main page of the provider documentation, PyPI README), will be automatically updated
to include removal notice.
For all those commands, release manager needs to specify ``--include-removed-providers`` when all providers
are built or must add the provider id explicitly during the release process.
Except the changelog that needs to be maintained manually, all other documentation (main page of the provider
documentation, PyPI README), will be automatically updated to include removal notice.
6 changes: 4 additions & 2 deletions airflow/providers/apache/sqoop/provider.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@ name: Apache Sqoop
description: |
`Apache Sqoop <https://sqoop.apache.org/>`__
suspended: false
source-date-epoch: 1700827444
removed: true
suspended: true
source-date-epoch: 1701983361
versions:
- 4.2.0
- 4.1.0
- 4.0.0
- 3.2.1
Expand Down
6 changes: 4 additions & 2 deletions airflow/providers/plexus/provider.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@ name: Plexus
description: |
`Plexus <https://plexus.corescientific.com/>`__
suspended: false
source-date-epoch: 1700827488
removed: true
suspended: true
source-date-epoch: 1701983408
versions:
- 3.4.0
- 3.3.0
- 3.2.2
- 3.2.1
Expand Down
6 changes: 3 additions & 3 deletions dev/breeze/tests/test_packages.py
Original file line number Diff line number Diff line change
Expand Up @@ -109,17 +109,17 @@ def test_get_provider_requirements():

def test_get_removed_providers():
# Modify it every time we schedule provider for removal or remove it
assert ["daskexecutor"] == get_removed_provider_ids()
assert ["apache.sqoop", "daskexecutor", "plexus"] == get_removed_provider_ids()


def test_get_suspended_provider_ids():
# Modify it every time we suspend/resume provider
assert ["daskexecutor"] == get_suspended_provider_ids()
assert ["apache.sqoop", "daskexecutor", "plexus"] == get_suspended_provider_ids()


def test_get_suspended_provider_folders():
# Modify it every time we suspend/resume provider
assert ["daskexecutor"] == get_suspended_provider_folders()
assert ["apache/sqoop", "daskexecutor", "plexus"] == get_suspended_provider_folders()


@pytest.mark.parametrize(
Expand Down
4 changes: 0 additions & 4 deletions docs/apache-airflow/extra-packages-ref.rst
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,6 @@ custom bash/python providers).
+---------------------+-----------------------------------------------------+------------------------------------------------+
| apache.spark | ``pip install 'apache-airflow[apache.spark]'`` | All Spark related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
| apache.sqoop | ``pip install 'apache-airflow[apache.sqoop]'`` | All Sqoop related operators & hooks |
+---------------------+-----------------------------------------------------+------------------------------------------------+
| apache.webhdfs | ``pip install 'apache-airflow[apache.webhdfs]'`` | HDFS hooks and operators |
+---------------------+-----------------------------------------------------+------------------------------------------------+

Expand Down Expand Up @@ -204,8 +202,6 @@ These are extras that add dependencies needed for integration with external serv
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
| pinecone | ``pip install 'apache-airflow[pinecone]'`` | Pinecone Operators and Hooks |
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
| plexus | ``pip install 'apache-airflow[plexus]'`` | Plexus service of CoreScientific.com AI platform |
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
| salesforce | ``pip install 'apache-airflow[salesforce]'`` | Salesforce hook |
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
| sendgrid | ``pip install 'apache-airflow[sendgrid]'`` | Send email using sendgrid |
Expand Down
15 changes: 0 additions & 15 deletions generated/provider_dependencies.json
Original file line number Diff line number Diff line change
Expand Up @@ -198,13 +198,6 @@
],
"excluded-python-versions": []
},
"apache.sqoop": {
"deps": [
"apache-airflow>=2.6.0"
],
"cross-providers-deps": [],
"excluded-python-versions": []
},
"apprise": {
"deps": [
"apache-airflow>=2.6.0",
Expand Down Expand Up @@ -751,14 +744,6 @@
"cross-providers-deps": [],
"excluded-python-versions": []
},
"plexus": {
"deps": [
"apache-airflow>=2.6.0",
"arrow>=0.16.0"
],
"cross-providers-deps": [],
"excluded-python-versions": []
},
"postgres": {
"deps": [
"apache-airflow-providers-common-sql>=1.3.1",
Expand Down
Loading

0 comments on commit 5422ceb

Please sign in to comment.