Status of testing Providers that were prepared on August 05, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.5.0rc1

• openlineage, sagemaker: add OpenLineage support for SageMaker's Processing, Transform and Training operators (#31816): @mobuchowski
• Add Amazon EventBridge PutRule hook and operator (#32869): @mahammi
• Add GCS Requester Pays bucket support to GCSToS3Operator (#32760): @hankehly
• Check google provider version in GCSToS3Operator before provide match_glob param (#32925): @hussein-awala
• FIX: set longer default waiter_max_attempts for deferred BatchJobOperator (#33045): @rishi-kulkarni
• openlineage, sagemaker: add missing OpenLineage type signature (#33114): @mobuchowski
• Add S3Bucket for mypy (#33028): @kimdwkimdw

Provider apache.drill: 2.4.3rc1

• Validate database URL passed to create_engine of Drill hook's connection (#33074): @pankajkoti

Provider apache.hive: 6.1.4rc1

• Bring back mysql-connector-python as required dependency (#32989): @potiuk

Provider apache.spark: 4.1.3rc1

• Validate conn_prefix in extra field for Spark JDBC hook (#32946): @pankajkoti

Provider apache.sqoop: 4.0.0rc1

• Validate SqoopHook connection string and disable extra options from public hook methods (#33039): @pankajkoti
• Validate connection host field for Sqoop connection (#32968): @pankajkoti
• Remove spurious headers for provider changelogs (#32373): @potiuk
• Improve provider documentation and README structure (#32125): @potiuk

Provider asana: 2.2.2rc1

• Limit Asana Python client until provider is adapted to 4.* version (#32995): @potiuk

Provider celery: 3.3.1rc1

• Add documentation generation for CLI commands from executors (#33081): @potiuk
• Get rid of Python2 numeric relics (#33050): @eumiro

Provider cncf.kubernetes: 7.4.1rc1

• Fix waiting the base container when reading the logs of other containers (#33127): @hussein-awala
• Fix: Configurable Docker image of xcom_sidecar (#32858): @pegasas
• Fix KPO sub classes default container_logs (#33090): @hussein-awala
• Consider custom pod labels on pod finding process on KubernetesPodOperator (#33057): @okayhooni
• Add documentation generation for CLI commands from executors (#33081): @potiuk

Provider common.sql: 1.6.2rc1

• Make SQLExecuteQueryOperator signature consistent with other SQL operators (#32974): @potiuk
• Get rid of Python2 numeric relics (#33050): @eumiro

Provider docker: 3.7.2rc1

• Get rid of Python2 numeric relics (#33050): @eumiro

Provider ftp: 3.5.0rc1

• openlineage, ftp: add OpenLineage support for FTPFileTransferOperator (#31354): @mobuchowski

Provider google: 10.6.0rc1

• openlineage, bigquery: add openlineage method support for BigQueryInsertJobOperator (#31293): @mobuchowski
• Add GCS Requester Pays bucket support to GCSToS3Operator (#32760): @hankehly
• Add system test and docs for CloudDataTransferServiceGCSToGCSOperator (#32960): @moiseenkov
• Fix names of resources in Datafusion system tests (#32855): @VladaZakharova
• Fix system test for MetastoreHivePartitionSensor (#32861): @moiseenkov

Provider imap: 3.3.0rc1

• Allows to choose SSL context for IMAP provider (#33108): @potiuk
• Add possibility to use ssl_context extra for SMTP and IMAP connections (#33112): @potiuk

Provider microsoft.azure: 6.2.3rc1

• Refactor account_url use in WasbHook (#32980): @ephraimbuddy

Provider mysql: 5.2.1rc1

• Bring back mysql-connector-python as required dependency (#32989): @potiuk

Provider openlineage: 1.0.1rc1

• openlineage: disable running listener if not configured (#33120): @mobuchowski
• openlineage: don't use database as fallback when no schema parsed. (#32959): @JDarDagran
• openlineage, bigquery: add openlineage method support for BigQueryInsertJobOperator (#31293): @mobuchowski
• Move openlineage configuration to provider (#33124): @potiuk

Provider redis: 3.3.1rc1

• Move redis subscribe to poke() method in Redis Sensor (#32984): @potiuk #33138

Provider smtp: 1.3.0rc1

• Add possibility to use ssl_context extra for SMTP and IMAP connections (#33112): @potiuk
• Allows to choose SSL context for SMTP provider (#33075): @potiuk

Provider snowflake: 4.4.1rc1

• Fix connection parameters of SnowflakeValueCheckOperator (#32605): @frodo2000

The guidelines on how to test providers can be found in

Verify providers by contributors

All users involved in the PRs:
@pankajkoti @moiseenkov @okayhooni @hankehly @frodo2000 @VladaZakharova @hussein-awala @mahammi @mobuchowski @kimdwkimdw @ephraimbuddy @potiuk @eumiro @pegasas @rishi-kulkarni @JDarDagran

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[rishi-kulkarni]

I can confirm that #33045 makes them default timeout sufficiently long.

[hankehly]

#32760 works on main branch (provider versions seem to match above).

Could someone please tell me how to install specific provider versions in a breeze environment? I am having some trouble following the directions in https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#provider-packages

[potiuk]

#32760 works on main branch (provider versions seem to match above).

Could someone please tell me how to install specific provider versions in a breeze environment? I am having some trouble following the directions in https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#provider-packages

See the link posted in the issue at the top Verify providers by contributors

There are very precise instructions.

UPDATE (but the link is not precise - pointing to old chapter - this is the right link :D https://github.com/apache/airflow/blob/main/dev/README_RELEASE_PROVIDER_PACKAGES.md#verify-the-release-candidate-by-contributors

[potiuk]

Checked all my changes - they all look good - except redis. I found out that the fix caused multiple "subscribe" calls for the sensor, so would likely cause problems. We should remove redis from the list, the fix for rc2 is coming.

[eladkal]

google and redis provider will be excluded from RC1 and will have rc2

PR to be included in RC2 of redis:
#33139

PR to be included in RC2 of google:
#33148

[mobuchowski]

Tested mine changes, they look good.

[hussein-awala]

Tested my changes, they are included and they work as expected

[kimdwkimdw]

Tested my changes with https://pypi.org/project/apache-airflow-providers-amazon/8.5.0rc1/

[okayhooni]

Tested my minor changes worked well
#33057

[ephraimbuddy]

Tested my refactor on azure #32980

[moiseenkov]

#32960 and #32861 work as expected

apache-airflow-providers-google==10.6.0rc1
apache-airflow-providers-celery==3.3.1rc1
apache-airflow-providers-amazon==8.5.0rc1
apache-airflow-providers-cncf-kubernetes==7.4.1rc1

[frodo2000]

#32605 <#32605> work as expected apache-airflow-providers-snowflake==4.4.1rc1 pon., 7 sie 2023 o 10:27 max ***@***.***> napisał(a):

…

#32960 <#32960> and #32861 <#32861> work as expected apache-airflow-providers-google==10.6.0rc1 apache-airflow-providers-celery==3.3.1rc1 apache-airflow-providers-amazon==8.5.0rc1 apache-airflow-providers-cncf-kubernetes==7.4.1rc1 — Reply to this email directly, view it on GitHub <#33132 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC7WCJL4VRGIH5TYC7ZJER3XUCRFPANCNFSM6AAAAAA3EXZR4Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[pankajkoti]

Tested #33074, #32946, #33039, #32968

Verified they are included in the RCs, ran tests and verified they work as expected.

[eladkal]

Thank you everyone.
Providers are released

google and redis providers are excluded, I will cut RC2 shortly

I invite everyone to help improve providers for the next release, a list of open issues can be found here.