Status of testing Providers that were prepared on July 29, 2023

[eladkal]

Body

I have a kind request for all the contributors to the latest provider packages release.
Could you please help us to test the RC versions of the providers?

Let us know in the comment, whether the issue is addressed.

Those are providers that require testing as there were some substantial changes introduced:

Provider amazon: 8.4.0rc1

• Add endpoint_url in test_connection (#32664): @ieunea1128
• adds support for querying Redshift Serverless clusters (#32785): @ivica-k
• Add Deferrable mode to StepFunctionStartExecutionOperator (#32563): @syedahsn
• Add Deferrable mode for EMR Serverless Start Job Operator (#32534): @syedahsn
• Add Eventbridge PutEvents operator and hook (#32498): @mahammi
• add deferrable mode to rds start & stop DB (#32437): @vandonr-amz
• EMR serverless Create/Start/Stop/Delete Application deferrable mode (#32513): @syedahsn
• Make Start and Stop SageMaker Pipelines operators deferrable (#32683): @vandonr-amz
• Deferrable mode for EKS Create/Delete Operator (#32355): @syedahsn
• FIX AWS deferrable operators by using AioCredentials when using assume_role (#32733): @rishi-kulkarni
• [bugfix] fix AWS triggers where deserialization would crash if region was not specified (#32729): @vandonr-amz
• Fix bug in prune_dict where empty dict and list would be removed even in strict mode (#32573): @vandonr-amz
• Fix S3ToRedshiftOperator does not support default values on UPSERT (#32558): @dashton90
• BUGFIX: AWS ECS trigger returned "success" after max_attempts instead of failing (#32589): @vandonr-amz
• Move all k8S classes to cncf.kubernetes provider (#32767): @potiuk

Provider apache.beam: 5.2.0rc1

• Add deferrable mode to BeamRunPythonPipelineOperator (#31471): @VladaZakharova

Provider apache.hive: 6.1.3rc1

• Fix Pandas2 compatibility for Hive (#32752): @potiuk
• add more accurate typing for DbApiHook.run method (#31846): @dwreeves
• Move Hive configuration to Apache Hive provider (#32777): @potiuk

Provider apache.pinot: 4.1.2rc1

• add more accurate typing for DbApiHook.run method (#31846): @dwreeves

Provider apache.spark: 4.1.2rc1

• Move all k8S classes to cncf.kubernetes provider (#32767): @potiuk

Provider apprise: 1.0.1rc1

• Replace Ruff's known-third-party setting with namespace-packages (#32873): @charliermarsh

Provider celery: 3.3.0rc1

• Move CeleryExecutor to the celery provider (#32526): @potiuk
• Add pre-Airflow-2-7 hardcoded defaults for config for older providers (#32775): @potiuk
• [AIP-51] Executors vending CLI commands (#29055): @o-nikolas
• Move all k8S classes to cncf.kubernetes provider (#32767): @potiuk
• Add Executors discovery and documentation (#32532): @potiuk
• Move default_celery.py to inside the provider (#32628): @potiuk
• Raise original import error in CLI vending of executors (#32931): @potiuk
• Introduce decorator to load providers configuration (#32765): @potiuk
• Allow configuration to be contributed by providers (#32604): @potiuk
• Remove spurious headers for provider changelogs (#32373): @potiuk
• Improve provider documentation and README structure (#32125): @potiuk

Provider cncf.kubernetes: 7.4.0rc1

• Move all k8S classes to cncf.kubernetes provider (#32767): @potiuk
• [AIP-51] Executors vending CLI commands (#29055): @o-nikolas
• Add termination_message_policy parameter to KubernetesPodOperator (#32885): @iJanki-gr
• Update the watcher resource version in SparkK8SOp when it's too old (#32768): @hussein-awala
• Raise original import error in CLI vending of executors (#32931): @potiuk

Provider common.sql: 1.6.1rc1

• Fix local OpenLineage import in SQLExecuteQueryOperator (#32400): @JDarDagran
• openlineage: add default port to authority method. (#32828): @JDarDagran
• add more accurate typing for DbApiHook.run method (#31846): @dwreeves

Provider daskexecutor: 1.0.0rc1

This is a new provider. Showing only the initial PR.

• Extract Dask executor to daskexecutor provider (#32772): @potiuk

Provider databricks: 4.3.2rc1

• Remove execute function of DatabricksRunNowDeferrableOperator (#32806): @Lee-W
• Add missing execute_complete method for DatabricksRunNowOperator (#32689): @Lee-W
• add more accurate typing for DbApiHook.run method (#31846): @dwreeves

Provider exasol: 4.2.2rc1

• add more accurate typing for DbApiHook.run method (#31846): @dwreeves

Provider google: 10.5.0rc1

• openlineage, gcs: add openlineage methods for GcsToGcsOperator (#31350): @mobuchowski
• Add Spot Instances support with Dataproc Operators (#31644): @VladaZakharova
• Install sqlalchemy-spanner package into Google provider (#31925): @MaksYermak
• Filtering and ordering results of DataprocListBatchesOperator (#32500): @kristopherkane
• Fix BigQueryGetDataOperator where project_id is not being respected in deferrable mode (#32488): @avinashpandeshwar
• Refresh GKE OAuth2 tokens (#32673): @fdemiane
• Fix BigQueryInsertJobOperator not exiting deferred state (#31591): @pankajastro
• add more accurate typing for DbApiHook.run method (#31846): @dwreeves
• Fix Datafusion system tests (#32749): @VladaZakharova

Provider microsoft.azure: 6.2.2rc1

• Add Redis task handler (#31855): @michalc

Provider mysql: 5.2.0rc1

• openlineage, mysql: add OpenLineage support for MySQL (#31609): @JDarDagran
• openlineage: add default port to authority method. (#32828): @JDarDagran

Provider neo4j: 3.3.2rc1

• add more accurate typing for DbApiHook.run method (#31846): @dwreeves

Provider openlineage: 1.0.0rc1

This is a new provider. Showing only the initial PR.

• Add base OpenLineage provider implementation (#29940): @mobuchowski

Provider postgres: 5.6.0rc1

• openlineage, postgres: add OpenLineage support for Postgres (#31617): @JDarDagran

Provider presto: 5.1.2rc1

• add more accurate typing for DbApiHook.run method (#31846): @dwreeves

Provider redis: 3.3.0rc1

• Add Redis task handler (#31855): @michalc

Provider sftp: 4.5.0rc1

• openlineage, sftp: add OpenLineage support for SFTPOperator (#31360): @mobuchowski

Provider slack: 7.3.2rc1

• add more accurate typing for DbApiHook.run method (#31846): @dwreeves

Provider snowflake: 4.4.0rc1

• openlineage, snowflake: add OpenLineage support for Snowflake (#31696): @JDarDagran
• add more accurate typing for DbApiHook.run method (#31846): @dwreeves

Provider trino: 5.2.1rc1

• add more accurate typing for DbApiHook.run method (#31846): @dwreeves

The guidelines on how to test providers can be found in

Verify providers by contributors

All users involved in the PRs:
@JDarDagran @fdemiane @ieunea1128 @kristopherkane @VladaZakharova @potiuk @Lee-W @hussein-awala @mobuchowski @charliermarsh @dwreeves @iJanki-gr @pankajastro @dashton90 @avinashpandeshwar @ivica-k
@o-nikolas @rishi-kulkarni @mahammi @vandonr-amz @michalc @MaksYermak @moiseenkov @syedahsn

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[ivica-k]

Tested the RedshiftDataOperator from the Amazon provider with this DAG:

from airflow import DAG
from airflow.providers.amazon.aws.operators.redshift_data import RedshiftDataOperator

default_args = {
    "start_date": "2023-03-01",
}

dag = DAG(
    "test_redshift_serverless",
    default_args=default_args,
    schedule="0 0 1,15 * *",
    catchup=False
)

try:
    rd = RedshiftDataOperator(
        task_id="run_this",
        dag=dag,
        database="dev",
        workgroup_name="ivica",
        sql="select current_user;",
        aws_conn_id="aws_default",
        wait_for_completion=True,
        return_sql_result=True
    )

    rd

except Exception as msg:
    print(msg)

Query result is:

{
	'ColumnMetadata': [{
		'isCaseSensitive': True,
		'isCurrency': False,
		'isSigned': False,
		'label': 'current_user',
		'length': 0,
		'name': 'current_user',
		'nullable': 1,
		'precision': 63,
		'scale': 0,
		'schemaName': '',
		'tableName': '',
		'typeName': 'bpchar'
	}],
	'Records': [
		[{
			'stringValue': 'IAM:ikolenkas'
		}]
	],
	'TotalNumRows': 1,
	'ResponseMetadata': {
		'RequestId': '8188e2dc-7129-4614-b630-36707a8a7669',
		'HTTPStatusCode': 200,
		'HTTPHeaders': {
			'x-amzn-requestid': '8188e2dc-7129-4614-b630-36707a8a7669',
			'content-type': 'application/x-amz-json-1.1',
			'content-length': '289',
			'date': 'Sat, 29 Jul 2023 15:05:51 GMT'
		},
		'RetryAttempts': 0
	}
}

To me this looks like a working operator, including the changes I added. Let me know if you expect a more thorough test.

[rishi-kulkarni]

I can confirm that deferrable operators work in combination with assume_role, per the fix I made in #32733. I tested this by using the BatchOperator with deferrable=True with a connection that uses STS to assume a role in another account.

[Lee-W]

We have tested the following providers with our example DAGS without encountering an issue.

apache-airflow-providers-amazon
apache-airflow-providers-apache-hive
apache-airflow-providers-cncf-kubernetes
apache-airflow-providers-databricks
apache-airflow-providers-google
apache-airflow-providers-snowflake
apache-airflow-providers-sftp
apache-airflow-providers-microsoft-azure

[potiuk]

Checked that all the changes I added are included - they import and install nicely. I've also run main airflow with the new providers installed from pypi - including running them with the Celery Executor and it seems that they are nicely working together.

I also added PR with instructions showing how easy it is to utilize Breeze features to run such tests - where you can combine for example "2.7.0dev0" airflow built from sources with RC candidates of few selected providers downloaded from PyPI. It's just few commands to execute #32948. Not everyone knows that you can get Airflow up and running (start-airflow with Release Candidate providers downloaded from PyPI in literally less than a minute):

[hussein-awala]

I checked #32768 and it's included in the RC. It's difficult to reproduce the error and test the fix.

[JDarDagran]

I checked my following changes in below providers:
apache-airflow-providers-common-sql==1.6.1rc1
apache-airflow-providers-mysql==5.2.0rc1
apache-airflow-providers-openlineage==1.0.0rc1
apache-airflow-providers-postgres==5.6.0rc1
apache-airflow-providers-snowflake==4.4.0rc1

by running example DAGs and seeing OpenLineage events in logs as per design.

[mobuchowski]

I've found one bug for #31350, fixing in: #32956

OpenLineage changes in sftp 4.5.0rc1 work as intended.
Will check more tomorrow.

[ieunea1128]

I've tested #32664 with following connection and there's no error with it.

conn = Connection(
    conn_id="sample_aws_connection",
    conn_type="aws",
    login="access_key",  
    password="secret_key", 
    extra={
        "test_endpoint_url": "https://sts.us-east-1.amazonaws.com",
        "region_name": "us-east-1"
    },
)

Also unit test is also added in #32664 to validate it.

[iJanki-gr]

#32885 works as expected.

[potiuk]

I've found one bug for #31350, fixing in: #32956

OpenLineage changes in sftp 4.5.0rc1 work as intended. Will check more tomorrow.

I guess it's not a blocker @mobuchowski ?

[mobuchowski]

@potiuk no, I believe it's not.

[fdemiane]

#32673 works as expected. The GKEStartPodOperator doesn't fail after the one hour mark anymore.
The following parameters were passed to the operator for testing: cmds=['sleep'], arguments=['4000']

[eladkal]

I've found one bug for #31350, fixing in: #32956

This doesn't seem critical and very locallized. I will not cut RC2 for this as this will delay Airflow 2.7.0 release further
It will wait for next provider release (we can have another wave next week)

[dashton90]

#32558 works as expected

[pankajastro]

#31591 Looking good. thanks!

[moiseenkov]

#31925 is working good

[vandonr-amz]

good for me

[moiseenkov]

Changes for #31471, #31644, #32749 working as expected, however some system tests are failing. We're fixing them.

apache-airflow-providers-amazon==8.4.0rc1
apache-airflow-providers-apache-beam==5.2.0rc1
apache-airflow-providers-celery==3.3.0rc1
apache-airflow-providers-cncf-kubernetes==7.4.0rc1
apache-airflow-providers-google==10.5.0rc1
apache-airflow-providers-postgres==5.6.0rc1
apache-airflow-providers-mysql==5.2.0rc1
apache-airflow-providers-ssh==3.7.1

[VladaZakharova]

Some problems found in #32749: incorrect naming of the resources lead to deleting resources before some action is actually finished.
This is fixed in #32855

[eladkal]

Some problems found in #32749: incorrect naming of the resources lead to deleting resources before some action is actually finished.

Can you please clarify what is the regression?
From the description it sounds like there is a bug which #32749 tried to solve but didn't completely and #32855 is the full fix.
Is this right? If this so.. this will not hold the release as there is no regression... the current RC did not make the problem worst. If this is accurate description I prefer to cut a followup release wave after this one is finished rather than having RC2.

[VladaZakharova]

Some problems found in #32749: incorrect naming of the resources lead to deleting resources before some action is actually finished.

Can you please clarify what is the regression? From the description it sounds like there is a bug which #32749 tried to solve but didn't completely and #32855 is the full fix. Is this right? If this so.. this will not hold the release as there is no regression... the current RC did not make the problem worst. If this is accurate description I prefer to cut a followup release wave after this one is finished rather than having RC2.

Yes, you are right here. Okay then, lets continue with current RC, i would expect #32855 in the next release. Thank you!

[eladkal]

Thank you everyone.
Providers are released
I invite everyone to help improve providers for the next release, a list of open issues can be found here.