gcs_to_bigquery using deprecated methods

[chenmic]

Hello,
I'm running Airflow 1.10.10 and using the GCSToBigQueryOperator from airflow.providers.google.cloud.transfers.gcs_to_bigquery in order to load data from google cloud storage into a big query table.
Although everything is working as expected, I keep seeing (a lot of) deprecation warnings in my logs:

DeprecationWarning: The bigquery_conn_id parameter has been deprecated. You should pass the gcp_conn_id parameter.
bigquery.py:103: DeprecationWarning: This method will be deprecated. Please use `BigQueryHook.get_client` method`
taskinstance.py:984: DeprecationWarning: This method is deprecated. Please use airflow.providers.google.cloud.hooks.bigquery.BigQueryHook.run_load`
bigquery.py:1614: DeprecationWarning: This method is deprecated. Please use `BigQueryHook.insert_job` method.

Although everything is working well, I'm afraid that in the future those warnings will indeed cause problems.
Please update the code and find a way to use the suggested methods instead of the deprecated ones.
If any more information is needed from me regarding the warnings or my system I will be happy to add it.
Thank you!

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[turbaszek]

Thanks for the issue @chenmic ! I think same goes for other BQ transfer operators as they were not refactored to use new methods.

[coopergillan]

Is this one anything of a priority?

[mik-laj]

This is a P4 level ticket. However, it is worth fixing this problem.

"P1" means Critical Impact – Service Unusable in Production
"P2" means High Impact – Service Use Severely Impaired
"P3" means Medium Impact – Service Use Partially Impaired
"P4" means Low Impact – Service Fully Usable

[coopergillan]

@mik-laj - I'm happy to take it as a P4. Plenty else to work on, but I can grab it when time allows.

[mik-laj]

@coopergillan I assigned you to this ticket.

[aprettyloner]

My team is also seeing the deprecation warning for decorator in gcs_to_bigquery on all GCSToBigQueryOperator imports. This is noisy and is misleading some of our developers to think we need to make a code change.

/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/transfers/gcs_to_bigquery.py:31 DeprecationWarning: This decorator is deprecated.

In previous versions, all subclasses of BaseOperator must use apply_default decorator for the`default_args` feature to work properly.

In current version, it is optional. The decorator is applied automatically using the metaclass.

[eladkal]

@aprettyloner are you willing to submit a PR to fix it?

[aprettyloner]

@aprettyloner are you willing to submit a PR to fix it?

Absolutely. Pretty tied in up migrating our DAGs from 1.10 to 2.1 (plus training new hires) but I am happy to open this once things settle down. I'll check on this issue status to ensure not duplicating work when the time comes. Thanks @eladkal

[eladkal]

@aprettyloner ping :)

[kazanzhy]

Hi all. @chenmic @aprettyloner
I've tried to remove warnings during GCSToBigQueryOperator as in PR #20119
But as a result, I copied almost all code from bq_hook.create_external_table and bq_hook.run_load which made the operator's execute method very sophisticated.
That's why I just switched from cursor method calls to BigQueryHook methods. In my opinion, these methods should not be deprecated.
We need someplace for complex dict creation, like configuration for bq_hook.insert_job() or table_recource for bq_hook.create_empty_table().
In my opinion, better to keep this code in BigQueryHook and don't move to Operator. But maybe I'm wrong.