Status of testing Providers that were prepared on April 07, 2022

[potiuk]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 3.3.0rc1

• Pass custom headers through in SES email backend (#22667): @repl-mike-roest
• Update secrets backends to use get_conn_value instead of get_conn_uri (#22348): @dstandish
• Add doc and sample dag for SqlToS3Operator (#22603): @vincbeck
• Adds HiveToDynamoDB Transfer Sample DAG and Docs (#22517): @ferruzzi
• Add doc and sample dag for MongoToS3Operator (#22575): @vincbeck
• Add doc for LocalFilesystemToS3Operator (#22574): @vincbeck
• Add doc and example dag for AWS CloudFormation Operators (#22533): @o-nikolas
• Add doc and sample dag for S3ToFTPOperator and FTPToS3Operator (#22534): @vincbeck
• GoogleApiToS3Operator: update sample dag and doc (#22507): @vincbeck
• SalesforceToS3Operator: update sample dag and doc (#22489): @vincbeck

Provider apache.livy: 2.2.3rc1

• Fix: Add extra headers to all livy API requests instead of only to post_batch (#22510): @cheelio

Provider celery: 2.1.4rc1

• Update our approach for executor-bound dependencies (#22573): @potiuk

Provider cncf.kubernetes: 4.0.0rc1

• Log traceback only on DEBUG for KPO logs read interruption (#22595): @dstandish
• Update our approach for executor-bound dependencies (#22573): @potiuk
• Optionally not follow logs in KPO pod_manager (#22412): @dstandish
• Stop crashing when empty logs are received from kubernetes client (#22566): @potiuk

Provider databricks: 2.6.0rc1

• More operators for Databricks Repos (#22422): @alexott
• Add an UI link to Databricks Job Run (#22541): @alexott
• Refactor DatabricksJobRunLink to not create ad hoc TaskInstances (#22571): @josh-fell

Provider discord: 2.1.4rc1

• DiscordWebhookOperator - add webhook_endpoint to templated fields (#22570): @ttomasz

Provider docker: 2.6.0rc1

• Add timeout parameter to DockerOperator (#22502): @Bowrna

Provider elasticsearch: 3.0.3rc1

• Make ElasticSearch Provider compatible for Airflow<2.3 (#22814): @kaxil

Provider google: 6.8.0rc1

• Add autodetect arg in BQCreateExternalTable Operator (#22710): @hankehly
• Add links for BigQuery Data Transfer (#22280): @wojsamjan
• Modify transfer operators to handle more data (#22495): @mwallace582
• Create Endpoint and Model Service, Batch Prediction and Hyperparameter Tuning Jobs operators for Vertex AI service (#22088): @MaksYermak
• PostgresToGoogleCloudStorageOperator - BigQuery schema type for time zone naive fields (#22536): @pierrejeambrun
• Update secrets backends to use get_conn_value instead of get_conn_uri (#22348): @dstandish
• Fix the docstrings (#22497): @mai-nakagawa
• Fix download_media url in GoogleDisplayVideo360SDFtoGCSOperator (#22479): @pierrejeambrun
• Fixed bug where CloudBuildRunBuildTriggerOperator fail to find build id. (#22419): @stephenoken
• Fail LocalFilesystemToGCSOperator if src does not exist (#22772): @pankajastro
• Remove coerce_datetime usage from GCSTimeSpanFileTransformOperator (#22501): @lwyszomi
• Refactor: BigQuery to GCS Operator (#22506): @shuhoy
• Remove references to deprecated operators/params in PubSub operators (#22519): @josh-fell
• New design of system tests (#22311): @mnojek

Provider hashicorp: 2.2.0rc1

• Update secrets backends to use get_conn_value instead of get_conn_uri (#22348): @dstandish

Provider jenkins: 2.1.0rc1

• Add JenkinsBuildSensor (#22421): @SasanAhmadi
• extra condition to check returned object not None (#22608): @SasanAhmadi

Provider microsoft.azure: 3.8.0rc1

• Update secrets backends to use get_conn_value instead of get_conn_uri (#22348): @dstandish
• Docs: Fix example usage for AzureCosmosDocumentSensor (#22735): @kaxil

Provider microsoft.psrp: 1.1.4rc1

• PowerShell Remoting fail on non-zero exitcode (#22503): @malthe

Provider presto: 2.2.0rc1

• Pass X-Presto-Client-Info in presto hook (#22416): @pingzh

Provider sftp: 2.6.0rc1

• Sftp Sensor Newer Than Parameter (#22377): @AdamPaslawski

Provider trino: 2.2.0rc1

• Pass X-Trino-Client-Info in trino hook (#22535): @pingzh

The guidelines on how to test providers can be found in

Verify providers by contributors

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[mnojek]

From my side the testing for Google was done :) #22311

[vincbeck]

All mine are doc and sample dag related so it's all good

[SasanAhmadi]

from my side:

• testing for jenkins sensor was done: #22421
• testing for jenkins job trigger operator was done: #22608

[shuhoy]

From my side, test run of BigQueryToGCSOperator was done. #22506

[raphaelauv]

I can't check the KPO 4.0.0rc1

because it ask for airlfow>=2.3.0rc1

Collecting apache-airflow-providers-cncf-kubernetes==4.0.0rc1
  Downloading apache_airflow_providers_cncf_kubernetes-4.0.0rc1-py3-none-any.whl (51 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.0/51.0 KB 807.9 kB/s eta 0:00:00
Requirement already satisfied: cryptography>=2.0.0 in /usr/local/lib/python3.9/site-packages (from apache-airflow-providers-cncf-kubernetes==4.0.0rc1->-r requirements-custom.txt (line 1)) (36.0.2)
Collecting kubernetes<24,>=21.7.0
  Downloading kubernetes-23.3.0-py2.py3-none-any.whl (1.5 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 713.0 kB/s eta 0:00:00
ERROR: Could not find a version that satisfies the requirement apache-airflow>=2.3.0rc1 (from apache-airflow-providers-cncf-kubernetes) (from versions: 1.10.9-bin, 1.8.1, 1.8.2rc1, 1.8.2, 1.9.0, 1.10.0, 1.10.1b1, 1.10.1rc2, 1.10.1, 1.10.2b2, 1.10.2rc1, 1.10.2rc2, 1.10.2rc3, 1.10.2, 1.10.3b1, 1.10.3b2, 1.10.3rc1, 1.10.3rc2, 1.10.3, 1.10.4b2, 1.10.4rc1, 1.10.4rc2, 1.10.4rc3, 1.10.4rc4, 1.10.4rc5, 1.10.4, 1.10.5rc1, 1.10.5, 1.10.6rc1, 1.10.6rc2, 1.10.6, 1.10.7rc1, 1.10.7rc2, 1.10.7rc3, 1.10.7, 1.10.8rc1, 1.10.8, 1.10.9rc1, 1.10.9, 1.10.10rc1, 1.10.10rc2, 1.10.10rc3, 1.10.10rc4, 1.10.10rc5, 1.10.10, 1.10.11rc1, 1.10.11rc2, 1.10.11, 1.10.12rc1, 1.10.12rc2, 1.10.12rc3, 1.10.12rc4, 1.10.12, 1.10.13rc1, 1.10.13, 1.10.14rc1, 1.10.14rc2, 1.10.14rc3, 1.10.14rc4, 1.10.14, 1.10.15rc1, 1.10.15, 2.0.0b1, 2.0.0b2, 2.0.0b3, 2.0.0rc1, 2.0.0rc2, 2.0.0rc3, 2.0.0, 2.0.1rc1, 2.0.1rc2, 2.0.1, 2.0.2rc1, 2.0.2, 2.1.0rc1, 2.1.0rc2, 2.1.0, 2.1.1rc1, 2.1.1, 2.1.2rc1, 2.1.2, 2.1.3rc1, 2.1.3, 2.1.4rc1, 2.1.4rc2, 2.1.4, 2.2.0b1, 2.2.0b2, 2.2.0rc1, 2.2.0, 2.2.1rc1, 2.2.1rc2, 2.2.1, 2.2.2rc1, 2.2.2rc2, 2.2.2, 2.2.3rc1, 2.2.3rc2, 2.2.3, 2.2.4rc1, 2.2.4, 2.2.5rc1, 2.2.5rc2, 2.2.5rc3, 2.2.5)
ERROR: No matching distribution found for apache-airflow>=2.3.0rc1

[potiuk]

@raphaelauv

I can't check the KPO 4.0.0rc1

Actually .. you can :) (see below).

because it ask for airlfow>=2.3.0rc1

That's cool and how it should be (except that it asks for airflow not airlfow that is :) (but typos are my own thing thay make far too ofen.

But to be cool - you actually can test it. You can install the provider in regular "main" breeze. The breeze in main is what soon will become v2-3-test branch and we will build 2.3.0 from it (and it is 2.3.0.dev version so it should be installable there).

Just remove sources of airflow/providers/cncf in Breeze/local sources (after starting a regular version of and do pip install apache-airflow-providers-cncf-kubernetes==4.0.0rc1 --no-deps and it should work.

[raphaelauv]

thanks @potiuk, I was not aware of --no-deps 👍

since Airflow ( without the kubernetesExectutor ) do not have Kubernetes as a dependency

it's possible to install

apache-airlfow==2.2.5
kubernetes==21.7.0
apache-airflow-providers-cncf-kubernetes==4.0.0rc1 --no-deps

Is there any feature of airflow 2.3 that restrict the usage of the kubernetes provider 4.0.X in airlfow 2.2.5 ?

---

otherwise #22412 -> WORK

[potiuk]

otherwise #22412 -> WORK

Is there any feature of airflow 2.3 that restrict the usage of the kubernetes provider 4.0.X in airlfow 2.2.5 ?

Yeah. The main problem is that Airlfow 2.2.5 has KubernetesExecutor that both - uses cncf.kubernetes provider to run AND relies on old < 20.0.0 of kubernetes library. A lot of things will work with kubernetes==21.7.0 - but there are quite a few things that won't work unfortunately. An example (but this is just an example) is here: #22560

The new provider depend on > 21 (for good reasons) and rather than risk that there are more things not working in Airflow 2.2.5 - we decided that 2.2.* line and before will have to stick with the old (well - not THAT old) kubernetes provider. Hopefully there will not be serious "issues" with the old provider (we can always release a critical fix for the the 3.* line of provider). But if you would like to use new features from kubernetes provider 4.0+, you will have to go Airflow 2.3+.

As a bonus, that might actually give people more incentive to move to Airlfow 2.3 at some point in time :).

[potiuk]

BTW. As a learning - in the future we might bundle K8s Executor with the provider, but that will require Airflow 3 and breaking changes I am afraid.

[ttomasz]

tested discord provider (#22570) and it works

[o-nikolas]

My changes are docs/example dags, so all good.

[hankehly]

#22710 (google 6.8.0rc1) works as expected on 2022/04/09

[pierrejeambrun]

Tested #22536 and working. I cannot test #22479 but looking good 👍

[pankajastro]

#22772: Tested (google 6.8.0rc1) working as expected

[alexott]

#22541 & #22571 both work. Found bug in #22422, fix will be today. So we'll need to cut rc2

[potiuk]

#22541 & #22571 both work. Found bug in #22422, fix will be today. So we'll need to cut rc2

Thanks.

Re #22422 - it depends on the bug - if it's a bug in new operators which can be wokrarounded or does not block things entirely, we can still release the rc1 and follow up in the next release of the provider. We'd only refrain from releasing a provider for sure if there is a serious regression.

[alexott]

@potiuk as result of this, CreateRepository operator doesn't work at all - really strange that I missed it when testing :-(
Opened #22885 to fix it.

[potiuk]

Ah.. Then yeah :) rc2 it is. Good that you tested !

[josh-fell]

#22519 is good! Just a few docstring changes but confirmed in the package files.

[repl-mike-roest]

Double checked #22667 validated custom headers are delivered in the resulting email delivery

[potiuk]

Thanks for testing everyone!

I proceed with the release of all providers except the Databricks one. RC2 for databricks follows.