-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix existing custom operators' templated fields that include logic checks #36484
Comments
I removed some of the template boilerplate from the description for clarity :) |
I believe all entries are fixed now? |
Almost, we still have #37519 open. |
I think it was fixed in another PR? |
I checked, and there's a small bug in the pre-commit logic that checks that the field's name is the same as the parameter's. It currently affects 6 files (including this one), I'll create a PR for that. I updated the list in the issue accordingly. |
It took a while, but the pre-commit is now operational with no exceptions :) |
🎉 🎉 🎉 🎉 🎉 🎉 |
Good job @shahar1 ! |
Introduction
PR #33786 aims to prevent templated field logic in
__init__
method of operators automatically by introducing a new pre-commit that validates each operator (see added limitations in the PR).In short, for templated fields, there should be only an assignment operation in the constructor (i.e.,
self.field = field
), and the name of the assigned field should be the same as the assigned parameter (i.e., the following is invalid:self.field = something_else
).Before merging this PR, we need to fix the existing operators in separate PRs that don't follow the new limitations.
The following list includes currently invalid operators by file paths that need to be fixed (plus details of what should be fixed).
It was manually scraped on Dec. 12 2023 from the PR's output of currently invalid operators' constructors.
Please feel free to create PRs to fix any of the files.
Tasks list
airflow/providers/apache/hive/transfers/hive_to_samba.py
airflow/providers/google/cloud/operators/dataproc.py
airflow/providers/apache/livy/operators/livy.py
LivyOperator's constructor lacks direct assignments for instance members corresponding to the following template fields (i.e., self.field_name = field_name or super.init(field_name=field_name,
...) ):
['spark_params']
LivyOperator's constructor contains invalid assignments to the following instance members that should be corresponding to template fields (i.e., self.field_name = field_name):
['self.spark_params']
airflow/providers/google/cloud/operators/bigquery.py
airflow/providers/google/cloud/transfers/bigquery_to_postgres.py
airflow/providers/alibaba/cloud/operators/analyticdb_spark.py
airflow/providers/microsoft/azure/operators/container_instances.py
airflow/providers/google/cloud/transfers/bigquery_to_mysql.py
airflow/providers/google/cloud/operators/cloud_storage_transfer_service.py
airflow/providers/google/cloud/operators/vertex_ai/auto_ml.py
airflow/providers/google/cloud/transfers/sftp_to_gcs.py
airflow/providers/amazon/aws/transfers/base.py
airflow/providers/amazon/aws/operators/datasync.py
airflow/providers/weaviate/operators/weaviate.py
airflow/providers/amazon/aws/transfers/gcs_to_s3.py
airflow/providers/amazon/aws/transfers/redshift_to_s3.py
airflow/providers/cncf/kubernetes/operators/pod.py
airflow/providers/amazon/aws/operators/eks.py
airflow/providers/apache/hive/operators/hive_stats.py
airflow/providers/google/cloud/operators/vertex_ai/custom_job.py
airflow/providers/google/cloud/operators/mlengine.py
airflow/providers/apache/spark/operators/spark_submit.py
airflow/providers/apache/spark/operators/spark_sql.py
airflow/providers/databricks/operators/databricks_sql.py
Code of Conduct
The text was updated successfully, but these errors were encountered: