Status of testing Providers that were prepared on July 27, 2021 (rc2)

[potiuk]

This is the second attempt (rc2) of releasing July Providers :). I carried over the "checkmarks" from the last testing (#17037) so there is no need to re-test those.

I have a kind request for all the contributors to the latest provider packages release.
Could you help us to test the RC versions of the providers and let us know in the comment, if the issue is addressed there.

Providers that need testing

Those are providers that require testing as there were some substantial changes introduced:

Provider airbyte: 2.1.0rc2

• Fixed wrongly escaped characters in amazon's changelog (#17020): @potiuk

Provider amazon: 2.1.0rc2

• Allow attaching to previously launched task in ECSOperator (#16685): @pmalafosse
• Update AWS Base hook to use refreshable credentials (#16770) (#16771): @baolsen
• Added select_query to the templated fields in RedshiftToS3Operator (#16767): @hewe
• AWS Hook - allow IDP HTTP retry (#12639) (#16612): @baolsen
• Update Boto3 API calls in ECSOperator (#16050): @scottypate
• Adding custom Salesforce connection type + SalesforceToS3Operator updates (#17162): @josh-fell
• Adding SalesforceToS3Operator to Amazon Provider (#17094): @josh-fell
• AWS DataSync Operator does not cancel task on Exception (#11011): @baolsen
• Fix wrong template_fields_renderers for AWS operators (#16820): @codenamestif
• AWS DataSync cancel task on exception (#11011) (#16589): @baolsen
• Fixed template_fields_renderers for Amazon provider (#17087): @oyarushe
• removing try-catch block to fix timeout exception getting ignored in aws batch operator (#17081): @ayushchauhan0811
• ECSOperator / pass context to self.xcom_pull as it was missing (when using reattach) (#17141): @pmalafosse
• Made S3ToRedshiftOperator transaction safe (#17117): @oyarushe

Provider apache.druid: 2.0.1rc2

• Fix error in Druid connection attribute retrieval (#17095): @potiuk

Provider apache.hive: 2.0.1rc2

• Add python 3.9 (#15515): @potiuk

Provider apache.spark: 2.0.0rc2

• Make SparkSqlHook use Connection (#15794): @uranusjr

Provider apache.sqoop: 2.0.1rc2

• Fix Minor Bugs in Apache Sqoop Hook and Operator (#16350): @ciancolo

Provider cncf.kubernetes: 2.0.1rc2

• BugFix: Using json string in template_field fails with K8s Operators (#16930): @kaxil

Provider dingding: 2.0.0rc2

• Fix hooks extended from http hook (#16109): @msumit

Provider discord: 2.0.0rc2

• Fix hooks extended from http hook (#16109): @msumit

Provider docker: 2.1.0rc2

• Adds option to disable mounting temporary folder in DockerOperator (#16932): @potiuk
• [FIX] Docker provider - retry docker in docker (#17061): @raphaelauv
• Fix string encoding in DockerOperator when using xcom / json (#13536): @AlessioM

Provider google: 5.0.0rc2

• apache-airflow-providers-google: google-ads-12.0.0 (#17111): @rv-irosario
• Google Ads Hook: Support newer versions of the google-ads library (#17160): @infused-kim
• Standardise dataproc location param to region (#16034): @Daniel-Han-Yang
• Adding custom Salesforce connection type + SalesforceToS3Operator updates (#17162): @josh-fell
• Update alias for field_mask in Google Memmcache (#16975): @potiuk
• fix: get_cluster call project_id as self.project_id (#17075): @iostreamdoth
• Fix GCStoGCS operator with replace diabled and existing destination object (#16991): @aslantar

Provider hashicorp: 2.0.0rc2

• Sanitize end of line character when loading token from a file (vault) (#16407): @mmenarguezpear

Provider http: 2.0.0rc2

• Update SimpleHttpOperator to take auth type object (#15605): @fredthomsen
• HttpHook. Use request factory and respect defaults (#14701): @ngaranko

Provider jenkins: 2.0.1rc2

• Fixed to check number key from jenkins response (#16963): @namjals

Provider microsoft.azure: 3.1.0rc2

• Add support for managed identity in WASB hook (#16628): @malthe
• WASB hook: reduce log messages for happy path (#16626): @malthe
• Fix multiple issues in Microsoft AzureContainerInstancesOperator (#15634): @BKronenbitter

Provider mysql: 2.1.0rc2

• Added template_fields_renderers for MySQL Operator (#16914): @oyarushe
• Extended template_fields_renderers for MySQL provider (#16987): @oyarushe
• Parse JSON template parameters field for MySQL operator (#17080): @oyarushe

Provider odbc: 2.0.0rc2

• OdbcHook returns None. Related to #15016 issue. (#15510): @Goodkat
• Fix OdbcHook handling of port (#15772): @dstandish

Provider opsgenie: 2.0.0rc2

• Fix hooks extended from http hook (#16109): @msumit

Provider oracle: 2.0.0rc2

• Add optional result handler to database hooks (#15581): @malthe
• [Oracle] Add port to DSN (#15589): @malthe
• Add link to Oracle Connection Docs (#15632): @sunkickr

Provider papermill: 2.0.0rc2

• Emit error on duplicated DAG ID (#15302): @uranusjr

Provider plexus: 2.0.0rc2

• Removes arrow higher limits for plexus provider (#16026): @potiuk

Provider postgres: 2.1.0rc2

• Add schema override in DbApiHook (#16521): @LukeHong

Provider qubole: 2.0.0rc2

• Qubole Hook Does Not Support 'include_headers' (#15598): @levyitay
• Feature qubole hook support headers (#15683): @levyitay
• Feature qubole hook support headers (#15615): @levyitay

Provider samba: 2.0.0rc2

• Add support for extra parameters to samba client (#16115): @malthe

Provider salesforce: 3.1.0rc2

• Adding custom Salesforce connection type + SalesforceToS3Operator updates (#17162): @josh-fell

Provider sftp: 2.1.0rc2

• Add support for non-RSA type client host key (#16314): @malthe

Provider slack: 4.0.0rc2

• Fix hooks extended from http hook (#16109): @msumit

Provider snowflake: 2.1.0rc2

• Adding: Snowflake Role in snowflake provider hook (#16735): @saurasingh

Provider ssh: 2.1.0rc2

• Add support for non-RSA type client host key (#16314): @malthe
• SSHHook: Using correct hostname for host_key when using non-default ssh port (#15964): @freget
• Correctly load openssh-gerenated private keys in SSHHook (#16756): @ashb

Provider tableau: 2.1.0rc2

• Allow disable SSL for TableauHook (#16365): @ciancolo
• Deprecate Tableau personal token authentication (#16916): @samgans
• Fix bool conversion Verify parameter in Tableau Hook (#17125): @ciancolo

New Providers

• apache.drill: 1.0.0rc1 @dzamo

[raphaelauv]

for docker 2.1.0rc2 it's all good

docker in docker work and also the XCOM

---

t3 = DockerOperator(
    api_version='auto',
    docker_url="unix://var/run/docker.sock",
    command='/bin/echo toto',
    image='centos:latest',
    network_mode='bridge',
    do_xcom_push=True,
    task_id='docker_op_tester',
    dag=dag,
)

give

[2021-07-27 19:38:44,647] {docker.py:246} INFO - Starting docker container from image centos:latest
[2021-07-27 19:38:44,682] {docker.py:256} WARNING - Using remote engine or docker-in-docker and mounting temporary volume from host is not supported. Falling back to `mount_tmp_dir=False` mode. You can set `mount_tmp_dir` parameter to False to disable mounting and remove the warning
[2021-07-27 19:38:46,975] {docker.py:307} INFO - toto
[2021-07-27 19:38:47,664] {taskinstance.py:1204} INFO - Marking task as SUCCESS. dag_id=docker_sample, task_id=docker_op_tester, execution_date=20210727T150000, start_date=20210727T193839, end_date=20210727T193847
[2021-07-27 19:38:47,770] {taskinstance.py:1265} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2021-07-27 19:38:47,815] {local_task_job.py:149} INFO - Task exited with return code 0

and

t3 = DockerOperator(
    api_version='auto',
    docker_url="unix://var/run/docker.sock",
    command='/bin/echo toto',
    image='centos:latest',
    network_mode='bridge',
    mount_tmp_dir=False,
    do_xcom_push=True,
    task_id='docker_op_tester',
    dag=dag,
)

give

[2021-07-27 19:39:27,176] {docker.py:246} INFO - Starting docker container from image centos:latest
[2021-07-27 19:39:28,091] {docker.py:307} INFO - toto
[2021-07-27 19:39:28,383] {taskinstance.py:1204} INFO - Marking task as SUCCESS. dag_id=docker_sample, task_id=docker_op_tester, execution_date=20210727T180000, start_date=20210727T193926, end_date=20210727T193928
[2021-07-27 19:39:28,424] {taskinstance.py:1265} INFO - 0 downstream tasks scheduled from follow-on schedule check
[2021-07-27 19:39:28,464] {local_task_job.py:149} INFO - Task exited with return code 0

and in all case a XCOM

key : return_value
value : toto

[levyitay]

qubole: 2.0.0rc2 tested and working

[josh-fell]

Contributions for Google, Amazon, and Salesforce providers look good!

[pmalafosse]

amazon: 2.1.0rc2:

• Allow attaching to previously launched task in ECSOperator (Allow attaching to previously launched task in ECSOperator #16685)
• ECSOperator / pass context to self.xcom_pull as it was missing (when using reattach) (ECSOperator / pass context to self.xcom_pull as it was missing (when using reattach) #17141)

I just tried it and ECSOperator is working as expected

[pmalafosse]

This one is fine too:

• Update Boto3 API calls in ECSOperator (Update Boto3 API calls in ECSOperator #16050)

[Goodkat]

Tested again the odbc connection, it works as expected with the boolean parameters:
#15016 (comment)

[potiuk]

Getting very good progress on testing with this release :) . Thanks a bunch for those! looking forward for more quick tests :)

[pavelhlushchanka]

#16820 is also good to go

[baolsen]

#16771 and #16612 tested and working as expected.

…

On Wed, 28 Jul 2021, 10:41 AM Pavel Hlushchanka, ***@***.***> wrote: #16820 <#16820> is also good to go — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#17268 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGLEIYZHSJOW72BLXQVJWYTTZ67CFANCNFSM5BCTM7EQ> .

[ayushchauhan0811]

#17081 is also good to go

[iostreamdoth]

fix: get_cluster call project_id as self.project_id (#17075) looks good.

[infused-kim]

Google ads #17160 is working fine, but I am the one who submitted the PR and it would be good to get it tested by a few more people.

[potiuk]

Google ads #17160 is working fine, but I am the one who submitted the PR and it would be good to get it tested by a few more people.

@Overbryd and @rv-irosario - since this was addressing your issues #17113 and #17111 - could you please also take a look if https://pypi.org/project/apache-airflow-providers-google/5.0.0rc2 solves the ads compatibility ?

[potiuk]

Hey @oyarushe @scottypate @uranusjr @msumit @Daniel-Han-Yang @rv-irosario @aslantar @malthe @LukeHong
@sunkickr @mmenarguezpear @fredthomsen @ngaranko @namjals @dstandish @BKronenbitter @freget @ashb @ciancolo @samgans - last chance to test before we release later tonight :)

[oyarushe]

I'm trying to check #16521, but the fix was provided in DbApiHook, which is a general class and I suppose will be released with Airflow itself.
So I think we need to release the Postgres provider after Airflow.

[potiuk]

So I think we need to release the Postgres provider after Airflow.

Ah yeah. This is backwards-compatible change, so it's ok to release Postgres now, but indeed testing will be possible only with new Airflow released (and it i's a new feature so it should be only released in 2.2. Good that you raised it!

[ciancolo]

Tested #16365 and #17125 on my local Tableau Server and they works.

[potiuk]

Thanks everyone who tested! This is really great help for the community. The vote has passed and I am proceeding to release the providers!