Fix BigQueryGetDataOperator where project_id is not being respected in deferrable mode #32488
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ct. A new parameter table_project_id will be used for specifying table storage project.
boring-cyborg
bot
added
area:providers
provider:google
avinashpandeshwar
changed the title
eladkal
reviewed
Jul 10, 2023
| will be returned from. If None, it will be derived from the hook's project ID. (templated) | ||
| :param table_project_id: (Optional) The project ID of the requested table. | ||
| If None, it will be derived from the hook's project ID. (templated) | ||
| :param project_id: (Optional) Google Cloud Project where the job is running. |
There was a problem hiding this comment.
You are changing what project_id means
If we want to do it can only be done in major release and even then its not something I am comfortable with as it may cause confusion for users.
My best advise here is deprecate project_id and use 2 new explicit parameter names.
There was a problem hiding this comment.
Thanks for the advice. I have reverted project_id's meaning and usage to as it was before. A new parameter job_project_id will handle job submission.
Lee-W
approved these changes
Jul 17, 2023
potiuk
approved these changes
Jul 20, 2023
This was referenced Jul 27, 2023
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
BigQueryGetDataOperator'sproject_idparameter is currently being used to specify table's storage project, but also used for job submissions in deferred mode. This PR aims to separate these responsibilities by introducing two new parameters:table_project_idto specify the table's storage project.job_project_idto specify job submission project to fetch data from the table.Both parameters are optional, and are defaulted by hook's project_id.
The pre-existing
project_idparameter is deprecated, with log warnings on usage.related: #32093
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.