Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor: BigQuery to GCS Operator #22506

Merged
merged 3 commits into from
Mar 27, 2022
Merged

Refactor: BigQuery to GCS Operator #22506

merged 3 commits into from
Mar 27, 2022

Conversation

shuhoy
Copy link
Contributor

@shuhoy shuhoy commented Mar 24, 2022

(Take over from #22252 and created new PR.)

Point to change

For simplicity and due to issue #22034 , change to use function run_extract in BigQueryHook to transfer execution.

Related issue

closes #22034


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@shuhoy shuhoy requested a review from turbaszek as a code owner March 24, 2022 14:00
@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Mar 24, 2022
@shuhoy
Copy link
Contributor Author

shuhoy commented Mar 26, 2022

All commits and checks are done.

@potiuk potiuk merged commit 02976be into apache:main Mar 27, 2022
@boring-cyborg
Copy link

boring-cyborg bot commented Mar 27, 2022

Awesome work, congrats on your first merged pull request!

@jedcunningham
Copy link
Member

@shuhoy, congrats on your first commit! In fact, you are the 2000th contributor to Airflow 🎉🚀!

2k contributors!

@potiuk
Copy link
Member

potiuk commented Mar 28, 2022

CONGRATS!

@shuhoy
Copy link
Contributor Author

shuhoy commented Mar 29, 2022

Oh! That's exciting!!
I'm grad to contribute to Airflow community!

@kaxil
Copy link
Member

kaxil commented Mar 29, 2022

Congratulations 🎉 -- well done :)

@gmyrianthous
Copy link

Any chance this is related to this issue?

@shuhoy
Copy link
Contributor Author

shuhoy commented Jun 8, 2022

Hi @gmyrianthous !
I think this PR is not related to your issue.
But I guess this change is related.
The following line in the error message you faced seems to have been added by this change.

  File "/opt/python3.8/lib/python3.8/site-packages/airflow/providers/google/cloud/transfers/bigquery_to_gcs.py", line 141, in execute
    job = hook.get_job(job_id=job_id).to_api_repr()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:providers provider:google Google (including GCP) related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BigQueryToGCSOperator: Invalid dataset ID error
5 participants