Discontinue using deprecated methods in GCSToBigQueryOperator #10354
Conversation
Several methods used in GCSToBigQueryOperator are deprecated. Switch to use the preferred methods, updating parameters and tests as needed.
boring-cyborg
bot
added
the
provider:google
|
@turbaszek and @mik-laj both chimed in over in the original issue #10288. |
| @@ -258,7 +258,7 @@ def execute(self, context): | |||
| cursor = conn.cursor() | |||
|
|
|||
| if self.external_table: | |||
| cursor.create_external_table( | |||
| bq_hook.create_external_table( | |||
There was a problem hiding this comment.
This method is also deprecated
| @@ -276,7 +276,7 @@ def execute(self, context): | |||
| encryption_configuration=self.encryption_configuration | |||
| ) | |||
| else: | |||
| cursor.run_load( | |||
| bq_hook.insert_job(configuration=dict( | |||
There was a problem hiding this comment.
That won't do:
airflow.exceptions.AirflowException: Unknown job type. Supported types: dict_keys(['load', 'copy', 'extract', 'query'])
as per docstring:
The configuration parameter maps directly to
BigQuery's configuration field in the job object. See
https://cloud.google.com/bigquery/docs/reference/v2/jobs for
details.
So what I would suggest is:
- add
configurationparameter to constructor - if this parameter is passed then use
insert_jobmethod - if not then show a deprecation warning that
Users should pass load job definitionand use the existing logic
Probably we should validate that only load job is passed... WDYT @edejong ?
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
stale
bot
added
the
stale
|
Hey @coopergillan do you have bandwitdth to continue this PR? It's quite important one as we have few issues around this operator 🔥 Maybe @juneoh would like to help? |
|
@turbaszek - sorry for the nearly-year-long silence. I am hhoping to take a look back into this as it appears that the issue #10288 is still open. |
Several methods used in
GCSToBigQueryOperatorare deprecated. Switch to use the preferred methods, updating parameters and tests as needed.related: #10288
Note: I only had time to tackle the operator mentioned by the user who opened the issue. It looks like the other Big Query transfer operators have the same issue. I could tackle those at a later date.
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.