Add ORC support to GCSToBigQueryOperator

[juneoh]

Description

BigQuery API for loading data lists ORC file format as one of the supported formats, but it is currently not allowed in GCSToBigQueryOperator. Specifying ORC as the source format raises a ValueError in BigQueryHook.

Use case / motivation

Include ORC file format in BigQueryHook so that GCSToBigQueryOperator will fully allow all file formats supported by the BigQuery API.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[turbaszek]

As mentioned in PR the run_load method is deprecated and users should use insert_job method which supports everything possible. That said, the change we should do is to update GCSToBigQueryOperator to use new method instead of the old one, see #10288 (the PR seems to be in there but needs some more attention)

[mik-laj]

@juneoh Are you willing to submit a PR?

[juneoh]

Sure, I'll take a look during the holidays.

[juneoh]

I've learned that holiday productivity is an illusion. I'll check in again next week as I'm ramping back up.

[juneoh]

Started working on this. Upon studying the history, looks like we need to fix the both paths in execute() for BigQuery and external tables to stop using the deprecated methods, which will likely to fix #10288 and #12329 along the way.

[juneoh]

Still working on this. The basic implementation is done but it's mostly a copy-paste from the BigQuery hook, so I'm trying to see what additional refactoring and testing is possible with the leverage of BigQuery Python SDK.

[eladkal]

@juneoh are you still working on it?

[eladkal]

The fix needed (while small) is in a deprecated function of the hook.
The function is deprecated for almost 2 years I believe we are closer to remove the function rather than fixing a bug in it.
Also no one tried to address this issue for very long time so I guess most users already moved away from the deprecated function.

I'm closing this issue as won't fix

[ericxiao251]

Hi @eladkal, I would love to take this issue up - I just faced this issue at work where we store all our data in ORC. I read the comments, briefly and understand that initially, the suggestions were to put the fix in a function/hook that is now deprecated.

What would be the approach to adding this feature today? Would you be able to point to the file/function in Airflow I should be making this change?

Seems like we will need to provide some of the same logical type mapping that is supported with AVRO as well: https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-orc#orc_conversions.