Status of testing Providers that were prepared on November 08, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

The guidelines on how to test providers can be found in

Verify providers by contributors

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.11.0rc1

• Add support for anonymous access to s3 buckets for objectstorage (#35273): @bolkedebruin
• ECS Executor Health Check (#35412): @o-nikolas
• Fix AWS RDS hook's DB instance state check (#34773): @AetherUnbound
• Fix parameter syntax in Amazon docstrings (#35349): @josh-fell
• Improve error handling in AWS Links (#35518): @Taragolis
• Update ECS executor healthcheck with a catchall except (#35512): @o-nikolas
• Move ECS Executor to its own file (#35418): @syedahsn
• Clarify "task" in ECS Executor log messages (#35304): @ferruzzi
• Make optional output_location attribute in AthenaOperator (#35265): @Taragolis

Provider apache.spark: 4.4.0rc1

• Add pyspark decorator (#35247): @bolkedebruin
• Add use_krb5ccache option to SparkSubmitOperator (#35331): @zeotuan

Provider cncf.kubernetes: 7.9.0rc1

• Add verification that provider docs are as expected (#35424): @potiuk
• Add startup_check_interval_seconds to PodManager's await_pod_start (#34231): @stelsemeyer-m60
• Remove before_log in KPO retry and add traceback when interrupted (#35423): @dstandish
• Remove tenacity on KPO logs inner func consume_logs (#35504): @dstandish
• Simplify KPO multi container log reconciliation logic (#35450): @dstandish
• Remove PodLoggingStatus object (#35422): @dstandish
• Improve clear_not_launched_queued_tasks call duration (#34985): @dirrao
• Use constant for empty xcom result sentinel (#35451): @dstandish

Provider cohere: 1.0.0rc1

• Add Cohere Provider (#34921): @utkarsharma2

Provider common.io: 1.0.1rc1

• fix changelog of common-io (#35241): @eladkal

Provider databricks: 5.0.0rc1

• Remove offset-based pagination from list_jobs function in DatabricksHook (#34926): @oleksiidav

Provider discord: 3.4.1rc1

• Add pagination to HttpOperator and make it more modular (#34669): @Joffreybvn

Provider docker: 3.8.1rc1

• fix _DockerDecoratedOperator module type attribute pickle error (#35293): @phi-friday

Provider elasticsearch: 5.1.1rc1

• Use None instead of empty data structures when no ElasticSearch logs (#34793): @dstandish

Provider ftp: 3.6.1rc1

• FTPHook methods should not change working directory (#35105): @makrushin-evgenii

Provider google: 10.11.1rc1

• Update Google Ads API version from v14 to v15 (#35295): @MaksYermak

Provider http: 4.7.0rc1

• Add pagination to HttpOperator and make it more modular (#34669): @Joffreybvn
• json data for async PUTs fixed (#35405): @andrew9691

Provider microsoft.azure: 8.2.0rc1

• add managed identity support to AsyncDefaultAzureCredential (#35394): @Lee-W
• add managed identity support to azure container registry hook (#35320): @Lee-W
• add managed identity support to wasb hook (#35326): @Lee-W
• add managed identity support to asb hook (#35324): @Lee-W
• add managed identity support to azure cosmos hook (#35323): @Lee-W
• add managed identity support to azure container volume hook (#35321): @Lee-W
• add managed identity support to azure container instance hook (#35319): @Lee-W
• add managed identity support to adx hook (#35325): @Lee-W
• add managed identity support to azure batch hook (#35327): @Lee-W
• add managed identity support to azure datalake hook (#35328): @Lee-W
• add managed identity support to synapse hook (#35329): @Lee-W
• add managed identity support to fileshare hook (#35330): @Lee-W
• Fix AzureContainerInstanceOperator remove_on_error (#35212): @danielvdende
• setting use_async=True for get_async_default_azure_credential (#35432): @Lee-W
• Remove empty TYPE_CHECKING block into the Azure provider (#35477): @Taragolis
• Refactor azure managed identity (#35367): @Lee-W
• Reuse get_default_azure_credential method from Azure utils for Azure key valut (#35318): @Lee-W
• make DefaultAzureCredential configurable in AzureKeyVaultBackend (#35052): @Lee-W
• Make DefaultAzureCredential in AzureBaseHook configuration (#35051): @Lee-W

Provider openai: 1.0.0rc1

• Add OpenAI Provider (#35023): @utkarsharma2

Provider openlineage: 1.2.1rc1

• openlineage, snowflake: fix Snowflake information schema query (#35428): @JDarDagran

Provider pgvector: 1.0.0rc1

• Add pgvector provider implementation (#35399): @pankajkoti

Provider pinecone: 1.0.0rc1

• Add Pinecone Provider (#35094): @utkarsharma2

Provider postgres: 5.8.0rc1

• Refactor cursor retrieval in PostgresHook. (#35498): @ylnsnv

Provider salesforce: 5.5.1rc1

• Fix SalesforceHook compatiblity with Pandas 2.x (#35145): @dave-pollock

Provider slack: 8.4.0rc1

• Reorganize SQL to Slack Operators (#35215): @Taragolis

Provider snowflake: 5.1.1rc1

• openlineage, snowflake: fix Snowflake information schema query (#35428): @JDarDagran
• Bump min snowflake-connector-python version (#35440): @romsharon98

Provider weaviate: 1.0.0rc1

• Add Weaviate Provider (#35060): @utkarsharma2

Provider yandex: 3.6.0rc1

• Yandex dataproc deduce default service account (#35059): @Piatachock

All users involved in the PRs:
@ferruzzi @ylnsnv @syedahsn @stelsemeyer-m60 @dave-pollock @eladkal @zeotuan @MaksYermak @AetherUnbound @pankajkoti @makrushin-evgenii @andrew9691 @utkarsharma2 @Joffreybvn @phi-friday @Taragolis
@JDarDagran @danielvdende @bolkedebruin @dstandish @dirrao @Lee-W @oleksiidav @josh-fell @romsharon98 @Piatachock @potiuk @o-nikolas

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

My change was only in "tests" and it works :)

[Lee-W]

Tested with default Azure credential change and a few examples dags. All look good 👍

[Piatachock]

✅ for Yandex provider

[JDarDagran]

#35428 tested, working as expected

[utkarsharma2]

+1 Tested my changes, working as expected. Thanks, Utkarsh Sharma

…

On Thu, Nov 9, 2023 at 5:46 PM Jakub Dardzinski ***@***.***> wrote: #35428 <#35428> tested, working as expected — Reply to this email directly, view it on GitHub <#35540 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADDLAHIYTFESXLAS33T25DLYDTCTRAVCNFSM6AAAAAA7DQXVD6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBTG4ZDGNBUGE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Joffreybvn]

There is something wrong with the HttpOperator. The new pagination functionality doesn't behave like expected with deferrable=False. But, it works fine when deferred.
I opened a PR with a fix #35560

[danielvdende]

Tested my change (35212), works as expected Daniel

…

On 9 Nov 2023 at 19:07 +0100, Joffrey Bienvenu ***@***.***>, wrote: There is something wrong with the HttpOperator. The new pagination functionality doesn't behave like expected with deferrable=False. But, it works fine when deferred. I opened a PR with a fix #35560 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: ***@***.***>

[MaksYermak]

Tested (35295), working as expected

[pankajkoti]

Tested #35399, working as expected. Thank you!

[AetherUnbound]

Tested #34773 locally - Sensors which were failing previously are now being rescheduled as expected! Looks good on my end!

[dirrao]

Tested #34985 locally- it is working as expected.

[phi-friday]

tested #35293 locally. it works as expected.

[eladkal]

Thank you everyone.
Providers are released (http and discord providers are excluded and RC2 will follow shortly)
I invite everyone to help improve providers for the next release, a list of open issues can be found here.