Status of testing Providers that were prepared on April 02, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider airbyte: 3.2.1rc1

• Clarify optional parameters in Airbyte docstrings (#30031): @josh-fell

Provider alibaba: 2.3.0rc1

• Support deleting the local log files when using remote logging (#29772): @hussein-awala

Provider amazon: 7.4.0rc1

• Add deferrable mode to RedshiftResumeClusterOperator (#30090): @phanikumv
• Add AwsToAwsBaseOperator (#30044): @eladkal
• Add deferrable mode in RedshiftPauseClusterOperator (#28850): @pankajastro
• Add support of a different AWS connection for DynamoDB (#29452): @1inuxoid
• Add EC2CreateInstanceOperator, EC2TerminateInstanceOperator (#29548): @syedahsn
• Make update config behavior optional in GlueJobOperator (#30162): @romibuzi
• custom waiters with dynamic values, applied to appflow (#29911): @vandonr-amz
• Support deleting the local log files when using remote logging (#29772): @hussein-awala
• Move string enum class to utils module + add test (#29906): @o-nikolas
• Align cncf provider file names with AIP-21 (#29905): @eladkal
• rewrite polling code for appflow hook (#28869): @vandonr-amz

Provider apache.drill: 2.3.2rc1

• Sanitize host in drill hook (#30215): @potiuk

Provider apache.hdfs: 3.2.1rc1

• Fix HDFSHook HAClient is invalid (#30164): @trickysky

Provider apache.hive: 6.0.0rc1

• Move auth parameter from extra to Hook parameter (#30212): @potiuk

Provider apache.livy: 3.4.0rc1

• Add non login-password auth support for SimpleHttpOpeator (#29206): @sirVir

Provider apache.spark: 4.0.1rc1

• Only restrict spark binary passed via extra (#30213): @potiuk
• Validate host and schema for Spark JDBC Hook (#30223): @potiuk
• Add spark3-submit to list of allowed spark-binary values (#30068): @ottomata

Provider cncf.kubernetes: 5.3.0rc1

• Enhance SparkKubernetesOperator (#29977): @blcksrx
• Fix KubernetesPodOperator xcom push when get_logs=False (#29052): @hussein-awala
• Fixed hanged KubernetesPodOperator (#28336): @moiseenkov
• Align cncf provider file names with AIP-21 (#29905): @eladkal
• Remove "boilerplate" from all taskflow decorators (#30118): @ashb
• Ensure setup/teardown work on a previously decorated function (#30216): @ephraimbuddy

Provider common.sql: 1.4.0rc1

• Add option to show output of SQLExecuteQueryOperator in the log (#29954): @eladkal
• Fix Python API docs formatting for Common SQL provider (#29863): @josh-fell

Provider databricks: 4.0.1rc1

• DatabricksSubmitRunOperator to support taskflow (#29840): @nivangio

Provider datadog: 3.2.0rc1

• Support creating Datadog connection from the UI and add documentation for this conn (#30059): @hussein-awala

Provider google: 8.12.0rc1

• Add missing poll_interval in Bigquery operator (#30132): @pankajastro
• Add poll_interval param in BigQueryInsertJobOperator (#30091): @pankajastro
• Add job_id to BigQueryToGCSOperator templated_fields (#30006): @eladkal
• Support deleting the local log files when using remote logging (#29772): @hussein-awala
• fix setting project_id for gs to bq and bq to gs (#30053): @Yaro1
• Fix location on cloud build operators (#29937): @tnk-ysk
• GoogleDriveHook: Fixing log message + adding more verbose documentation (#29694): @aru-trackunit
• Add "BOOLEAN" to type map of MSSQLToGCSOperator and fix incorrect BIT->INT conversion (#29902): @shahar1
• Google Cloud Providers - Fix _MethodDefault deepcopy failure (#29518): @IKholopov
• Handling project location param on async BigQuery dts trigger (#29786): @okayhooni
• Support CloudDataTransferServiceJobStatusSensor without specifying a project_id (#30035): @aibazhang
• Fix BigQueryToGCSOperator by waiting insert_job result in normal mode (#29925): @hussein-awala
• merge BigQueryTableExistenceAsyncSensor into BigQueryTableExistenceSensor (#30235): @Lee-W
• Remove 2 outdated max constraints in Google provider (#29915): @r-richmond
• Merge BigQueryTableExistencePartitionAsyncSensor into BigQueryTableExistencePartitionSensor (#30231): @Lee-W
• Merge GCSObjectExistenceAsyncSensor logic to GCSObjectExistenceSensor (#30014): @Lee-W
• Align cncf provider file names with AIP-21 (#29905): @eladkal
• Switch to using vendored-in google ads. (#30410): @potiuk
• Merging of the google ads vendored-in code. (#30399): @potiuk

Provider hashicorp: 3.3.1rc1

• Improve creating Vault connection from the UI and add documentation for this conn (#30057): @hussein-awala
• Remove required validations in Hashicorp Vault conn form (#30138): @josh-fell
• Remove unused key_id variable (#30055): @hussein-awala
• Simplify **dict (#30060): @uranusjr

Provider http: 4.3.0rc1

• Add non login-password auth support for SimpleHttpOpeator (#29206): @sirVir

Provider jenkins: 3.2.1rc1

• JenkinsHook fix connection intake on https (#30301): @vchiapaikeo

Provider microsoft.azure: 5.3.0rc1

• Add deferrable AzureDataFactoryRunPipelineOperator (#30147): @phanikumv
• Add deferrable AzureDataFactoryPipelineRunStatusSensor (#29801): @phanikumv
• Support deleting the local log files when using remote logging (#29772): @hussein-awala
• Fix ADF job failure during deferral (#30248): @phanikumv
• Fix AzureDataLakeStorageV2Hook with Active Directory authentication (#29981): @adamwhitneysoftwire
• merge AzureDataFactoryPipelineRunStatusAsyncSensor to AzureDataFactoryPipelineRunStatusSensor (#30250): @Lee-W
• Expose missing params in AzureSynapseHook API docs (#30099): @josh-fell

Provider ssh: 3.6.0rc1

• SSHOperator - Restore ability to override SSHHook cmd_timeout (#30190): @punx120

The guidelines on how to test providers can be found in

Verify providers by contributors

All users involved in the PRs:
@Lee-W @IKholopov @uranusjr @josh-fell @vandonr-amz @aru-trackunit @o-nikolas @1inuxoid @pankajastro @syedahsn @tnk-ysk @trickysky @vchiapaikeo @r-richmond @blcksrx @moiseenkov @punx120 @nivangio @ashb
@sirVir @romibuzi @shahar1 @Yaro1 @phanikumv @ottomata @hussein-awala @ephraimbuddy @eladkal @adamwhitneysoftwire @potiuk @aibazhang @okayhooni

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[potiuk]

Verified that my changes are present in the new providers:

• Move auth parameter from extra to Hook parameter #30212, Sanitize host in drill hook #30215 - hive/drill sanitizations
• Only restrict spark binary passed via extra #30213, Validate host and schema for Spark JDBC Hook #30223 Add spark3-submit to list of allowed spark-binary values #30068 - spark (though @ottomata I would really love if you can test it - especially that it not only allows to use spark3-submit as binary, but also restores the possibility to choose arbitraty binary via Operator/Hook rather than connection extra
• Switch to using vendored-in google ads. #30410 and Merging of the google ads vendored-in code. #30399 -> I see the vendored-in google-ads library is nicely included and used, but I would REALLY appreciate if @collinmcnulty and @cgadam to help with testing if it solves the problem described in Google Ads API Sunset for Version 11 #30353

[hussein-awala]

I tested:

• Improve creating Vault connection from the UI and add documentation for this conn #30057 - the new widgets are added to the UI and the connection works as expected
• Fix KubernetesPodOperator xcom push when get_logs=False #29052 - I tried simulating the problem locally, and the fix seemed to work
• Support creating Datadog connection from the UI and add documentation for this conn #30059 - the Datadog connection is added to the UI, I've created a real one, and it works without any problem
• Remove unused key_id variable #30055 - it's included in the new version
• Fix BigQueryToGCSOperator by waiting insert_job result in normal mode #29925 - works as expected

For #29772 I will test it tomorrow for Amazon provider, but I can't test it for other providers.

[vchiapaikeo]

Tested #30301 and it still looks good!

[phanikumv]

Is it possible to include #30370 in the amazon RC ? It fixes a bug in #30090.

[aibazhang]

I tested #30035, it works as expected

[eladkal]

Is it possible to include #30370 in the amazon RC ? It fixes a bug in #30090.

No. this will be released in next wave

[adamwhitneysoftwire]

Tested Fix AzureDataLakeStorageV2Hook with Active Directory authentication #29981 - The connection form improvements were present and the connection worked as expected.

[aru-trackunit]

Tested #29694 - works as expected

[blcksrx]

Works as expected #29977

[ephraimbuddy]

#30216 should not be included in a release yet. Is there a way this can be avoided @eladkal

[potiuk]

#30216 should not be included in a release yet. Is there a way this can be avoided @eladkal

Good catch, but I think this is a wrong question.

Part of the change is in cncf.kubernetes @ephraimbuddy -> this is what''s being released, so if it is not compatible with currently released airflow, it should be fixed. Generally the expectation is that if there is any change in a provider that require some changes to airflow core, one of the two things happen:

a) it is made in a backwards-compatble way that will be well handled in the older airflow version (by fallback, etc.)
b) it limits min-airflow version to be the future airflow version in provider.yaml

In this case, I think a) case is the right approach, so my question is (and mostly to you @ephraimbuddy :)

Is it backwards compatible with airlfow 2.3+?

if yes-> all is cool and we can release cncnf.kubernetes as is. If not, we should remove cncf.kubernetes from the release and fix it so that it is backwards compatible (and prepare rc2).

[potiuk]

if yes-> all is cool and we can release cncnf.kubernetes as is. If not, we should remove cncf.kubernetes from the release and fix it so that it is backwards compatible (and prepare rc2).

BTW. Looking at the code, it seems it is backwards-compatible, but I would love if you can confirm it @ephraimbuddy. And if it is, there is absolutely no problem in including that change in this release and it is how it should be. Imagine someone installing 2.6.0 tomorrow with setup teardown. If there is no released version of cncf.kubernetes that is compatible with it released, then we would be in a big trouble - this actueally means that we HAVE TO release cncf.kubernetes with this change before Airflow 2.6 is released. There is no other way. If we don't release it before, it will not work properly when 2.6.0 is released.

[punx120]

Tested #30190, works as expected

[romibuzi]

Tested #30162, works as expected

[ephraimbuddy]

#30216 should not be included in a release yet. Is there a way this can be avoided @eladkal

Good catch, but I think this is a wrong question.

Part of the change is in cncf.kubernetes @ephraimbuddy -> this is what''s being released, so if it is not compatible with currently released airflow, it should be fixed. Generally the expectation is that if there is any change in a provider that require some changes to airflow core, one of the two things happen:

a) it is made in a backwards-compatble way that will be well handled in the older airflow version (by fallback, etc.) b) it limits min-airflow version to be the future airflow version in provider.yaml

In this case, I think a) case is the right approach, so my question is (and mostly to you @ephraimbuddy :)

Is it backwards compatible with airlfow 2.3+?

if yes-> all is cool and we can release cncnf.kubernetes as is. If not, we should remove cncf.kubernetes from the release and fix it so that it is backwards compatible (and prepare rc2).

Yeah. It’s backwards compatible. I discussed with @eladkal and he said the release doesn’t include the test files and it’s only in the test files that we made import that should have caused backwards incompatibility.

[potiuk]

Yeah. It’s backwards compatible. I discussed with @eladkal and he said the release doesn’t include the test files and it’s only in the test files that we made import that should have caused backwards incompatibility.

Cool! Great you double checked it!

[phanikumv]

Tested the below PR and they are working fine

#30147
#29801
#30248

[josh-fell]

#30031, #30099, #29863, and #30138 look good.

Thanks for organizing @eladkal!

[tnk-ysk]

@eladkal
Testing #29937, I found another bug related to this fix and opened PR #30441.
This bug was not caused by #29937, but due to that, I have not been able to confirm success of deferrable.
Other fix seem fine.

[eladkal]

thank you @tnk-ysk
since this is not a regression I tend to continue release of the provider as is.
@potiuk WDYT ?

[potiuk]

Yes. Not regression. But I also have some news about the "ads" - we have found with @cgadam that there are still some problems after vendoring in, and we iterate together on a fix and it might mean either a quickl rc2 (if we do it before the relese end) or 8.12.1 release very quickly.

[eladkal]

Im in favor of 8.12.1 when we are ready with it.

[potiuk]

Sure. No problem with that (and I agree) . BTW. we are VERY close I think :).

[vandonr-amz]

good for the appflow changes for me

[o-nikolas]

The enum changes are low impact and the ECS system test has been passing with the new code. Looks all clear to me 👍

[hussein-awala]

I tested #29772 with S3TaskHandler and it works as expected.

Since it need an Airflow version >= 2.6.0, I had to create a subclass from S3TaskHandler with delete_local_copy=True and use it for task logs.

[syedahsn]

The EC2CreateInstanceOperator and EC2TerminateInstanceOperator are being used in our system tests, and have been passing.

[tnk-ysk]

Thank you @eladkal
I tested again using the GOOGLE_APPLICATION_CREDENTIALS environment.
#29937 looks good.

[pankajastro]

#28850 #30132 #30091 looks good. thank you!

[eladkal]

Thank you everyone.
Providers are released
I invite everyone to help improve providers for the next release, a list of open issues can be found here.