-
Notifications
You must be signed in to change notification settings - Fork 306
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: make QueryJob.done() method more performant #544
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
plamut
added
the
do not merge
Indicates a pull request not ready for merge, due to either quality or timing.
label
Mar 9, 2021
google-cla
bot
added
the
cla: yes
This human has signed the Contributor License Agreement.
label
Mar 9, 2021
product-auto-label
bot
added
the
api: bigquery
Issues related to the googleapis/python-bigquery API.
label
Mar 9, 2021
plamut
removed
the
do not merge
Indicates a pull request not ready for merge, due to either quality or timing.
label
Mar 10, 2021
plamut
requested review from
a team,
stephaniewang526 and
tswast
and removed request for
a team
March 10, 2021 14:09
tswast
approved these changes
Mar 10, 2021
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Magnificent!
gcf-merge-on-green bot
pushed a commit
that referenced
this pull request
Mar 16, 2021
🤖 I have created a release \*beep\* \*boop\* --- ## [2.12.0](https://www.github.com/googleapis/python-bigquery/compare/v2.11.0...v2.12.0) (2021-03-16) ### Features * make QueryJob.done() method more performant ([#544](https://www.github.com/googleapis/python-bigquery/issues/544)) ([a3ab9ef](https://www.github.com/googleapis/python-bigquery/commit/a3ab9efdd0758829845cfcb6ca0ac1f03ab44f64)) ### Bug Fixes * remove DB-API dependency on pyarrow with decimal query parameters ([#551](https://www.github.com/googleapis/python-bigquery/issues/551)) ([1b946ba](https://www.github.com/googleapis/python-bigquery/commit/1b946ba23ee7df86114c6acb338ec34e6c92af6d)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
This was referenced Mar 17, 2021
Merged
Closed
Closed
Closed
emkornfield
pushed a commit
to emkornfield/python-bigquery
that referenced
this pull request
Feb 11, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
api: bigquery
Issues related to the googleapis/python-bigquery API.
cla: yes
This human has signed the Contributor License Agreement.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #534.
This PR removes refreshing query results from the
QueryJob.done()
method, the latter is now just thedone()
method inherited from the_AsyncJob
base class that at most reloads the job itself and checks if its state is DONE.Since blocking poll from the
PollingFuture
base class repeatedly invokesdone()
, the change would cause too many job reload requests while waiting for the query results. TheQueryJob
class thus overrides the_done_or_raise()
method repeatedly used by the blocking poll so that the polling is actually performed by fetching the query results. The latter call can block for up to 10 seconds, meaning that fewer polling requests are made than if reload the job was used.How to test
Set logging level to DEBUG to see what HTTP requests are made. Then run a query job that normally takes more than 10 seconds to complete, for example:
While the query is running, call
query_job.result()
- query results should be fetched, but with a reasonable amount of requests.Beside testing the
.result()
method, the.done()
method should be checked, too - if it is run repeatedly while the query is running, each call should finish "fast", i.e. it should block for significantly less time than 10 seconds, because all it needs to do is to reload the job data itself.PR checklist: