feat: reconfigure tqdm progress bar in %%bigquery magic #1355

aribray · 2022-09-15T17:33:19Z

Change-Id: I2add62e3cdd5f25f88ace2d08f212796918158b6

Adds the BigQuery job ID to the tqdm progress bar description for a more helpful user message. Adds the correct tqdm.notebook import. Changes the default progress bar to tqdm_notebook

fixes: #1378
See internal 242869795

Change-Id: I2add62e3cdd5f25f88ace2d08f212796918158b6

google/cloud/bigquery/_tqdm_helpers.py

Change-Id: I6c4001608af1bd8c305c53c6089d64f99605bd8c

Change-Id: I2627d3ee386b59abd427d2837c83e9f92ec1f35b

Change-Id: I5788448d580b53898e75fba68ff5d5a9d12e33d6

Change-Id: I87e45085b7535083327a5fe2e51dba4b6411db00

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

Change-Id: Ibe0fc01db05fcfaacdbe0c074b841ead3a39afc9

…to aribray--tqdm Change-Id: I6448d5fb1d6a4e91a405b93b099ac00d748087d8

Change-Id: I56f8f98853e83ead0e0ca743c03407a521370233

Change-Id: I2d55e529142ad0024ef4a98c2f15d10a73535380

google/cloud/bigquery/magics/magics.py

Change-Id: I7961ff1c5e9c54930d077e67ef9e01d79e351c5f

Change-Id: I183e277fc7be8797c85d6802f4f8c3947871d4cc

Change-Id: I3b4a1b9460227ca49bf344362efbcc2c895d804d

…y--tqdm

shollyman

One minor question about the helpers, otherwise LGTM.

google/cloud/bigquery/_tqdm_helpers.py

tswast

Looks great! Thanks so much for working through those testing issues. Just one nit, that might be worth a little follow up

tswast · 2022-10-12T14:15:49Z

google/cloud/bigquery/_tqdm_helpers.py

            )
        try:
            query_result = query_job.result(
                timeout=_PROGRESS_BAR_UPDATE_INTERVAL, max_results=max_results
            )
            progress_bar.update(default_total)
            progress_bar.set_description(
-                "Query complete after {:0.2f}s".format(time.time() - start_time),
+                f"Job ID {query_job.job_id} successfully executed",


Might want to include the location in the output too, formatted like the bq cli as location:I'd https://cloud.google.com/python/docs/reference/bigquery/latest/google.cloud.bigquery.job.QueryJob#google_cloud_bigquery_job_QueryJob_location

I believe the location is required to fetch the job metadata for the non-default location.

chalmerlowe

A couple of minor tweaks and a question.
Other than that, looks good to me.

chalmerlowe · 2022-10-12T14:40:42Z

google/cloud/bigquery/magics/magics.py

@@ -286,7 +286,7 @@ def _handle_error(error, destination_var=None):

    Args:
        error (Exception):
-            An exception that ocurred during the query exectution.
+            An exception that ocurred during the query execution.


Change the word ocurred to the correct spelling of occurred

An exception that occurred during the query execution.

chalmerlowe · 2022-10-12T14:52:32Z

tests/system/test_magics.py

@@ -71,8 +71,7 @@ def test_bigquery_magic(ipython_interactive):
    # Removes blanks & terminal code (result of display clearing)
    updates = list(filter(lambda x: bool(x) and x != "\x1b[2K", lines))
    assert re.match("Executing query with job ID: .*", updates[0])
-    assert all(re.match("Query executing: .*s", line) for line in updates[1:-1])
-    assert re.match("Query complete after .*s", updates[-1])
+    assert (re.match("Query executing: .*s", line) for line in updates[1:-1])


Using assert with a tuple will always return True, no matter what is in the tuple unless the tuple is empty.
Are we trying to detect whether any matches are True OR are we trying to detect whether ALL the responses are True (...as implied by the deleted code above)?

In [9]: assert (True, True, True) <>:1: SyntaxWarning: assertion is always true, perhaps remove parentheses? <ipython-input-9-0a5188c9e554>:1: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert (True, True, True) In [10]: assert (False, False, False) <>:1: SyntaxWarning: assertion is always true, perhaps remove parentheses? <ipython-input-10-f31b4c57c272>:1: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert (False, False, False) In [11]: assert (False, False, True) <>:1: SyntaxWarning: assertion is always true, perhaps remove parentheses? <ipython-input-11-59e8f7bc6fa4>:1: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert (False, False, True) In [14]: assert (True,) <>:1: SyntaxWarning: assertion is always true, perhaps remove parentheses? <ipython-input-14-7b46914816ee>:1: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert (True,) In [15]: assert (False,) <>:1: SyntaxWarning: assertion is always true, perhaps remove parentheses? <ipython-input-15-a62a5cb79bea>:1: SyntaxWarning: assertion is always true, perhaps remove parentheses? assert (False,) In [16]: assert () --------------------------------------------------------------------------- AssertionError Traceback (most recent call last) Input In [16], in <cell line: 1>() ----> 1 assert ()

) * feat: add bigquery job id to tqdm progress bar description Change-Id: I2add62e3cdd5f25f88ace2d08f212796918158b6 * write to sys.stdout instead of sys.stderr Change-Id: I6c4001608af1bd8c305c53c6089d64f99605bd8c * configure progress bar Change-Id: I5788448d580b53898e75fba68ff5d5a9d12e33d6 * tqdm.notebook Change-Id: I87e45085b7535083327a5fe2e51dba4b6411db00 * 🦉 Updates from OwlBot post-processor See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md * reinclude ipywidgets Change-Id: Ibe0fc01db05fcfaacdbe0c074b841ead3a39afc9 * reinclude ipywidgets Change-Id: I56f8f98853e83ead0e0ca743c03407a521370233 * change test assertions to tqdm_notebook Change-Id: I2d55e529142ad0024ef4a98c2f15d10a73535380 * change test assertions in test_magics Change-Id: I7961ff1c5e9c54930d077e67ef9e01d79e351c5f * remove ipywidgets Change-Id: I183e277fc7be8797c85d6802f4f8c3947871d4cc * update assertions in test Change-Id: I3b4a1b9460227ca49bf344362efbcc2c895d804d * update method args in query.py and table.py Change-Id: I9a2bf2b54579668ff36ed992e599f4c7fabe918c * string formatting * fix typo * fix incorrect import structure for tqdm notebook * change default decorator back to tqdm * modify system test * add ipywidgets package for tqdm.notebook feature, set tqdm.notebook as default decorator for bq magic * change test assertion in test_query_pandas * revert test changes * reformat import statement * reformat import statement * remove timeouterror side effect * add tqdm mock patch * Revert "reformat import statement" This reverts commit 4114221. * Revert "add tqdm mock patch" This reverts commit ef809a0. * add timeout side effect * fix assertion * fix import * change mock patch to tqdm * move assertion * move assertions * add timeout side effect * adjust import statement, mock.patch tqdm * create fixture * revert import change * add import from helper * fix linting * remove unused imort * set ipywidgets version to 7.7.1 * set ipywidgets version to 7.7.1 * set ipywidgets version to 7.7.1 * bump sphinx version * bump sphinx version Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

feat: add bigquery job id to tqdm progress bar description

ac10e76

Change-Id: I2add62e3cdd5f25f88ace2d08f212796918158b6

product-auto-label bot added size: xs Pull request size is extra small. api: bigquery Issues related to the googleapis/python-bigquery API. labels Sep 15, 2022

aribray marked this pull request as ready for review September 15, 2022 18:07

aribray requested a review from a team September 15, 2022 18:07

aribray requested a review from a team as a code owner September 15, 2022 18:07

aribray requested a review from alvarowolfx September 15, 2022 18:07

aribray self-assigned this Sep 15, 2022

chalmerlowe reviewed Sep 19, 2022

View reviewed changes

google/cloud/bigquery/_tqdm_helpers.py Outdated Show resolved Hide resolved

write to sys.stdout instead of sys.stderr

9db15fc

Change-Id: I6c4001608af1bd8c305c53c6089d64f99605bd8c

aribray changed the title ~~feat: add bigquery job id to tqdm progress bar description~~ feat: reconfigure tqdm progress bar in %%bigquery magic Sep 23, 2022

aribray added 2 commits September 23, 2022 08:26

Merge remote-tracking branch 'upstream/main' into aribray--tqdm

6ed2f03

Change-Id: I2627d3ee386b59abd427d2837c83e9f92ec1f35b

configure progress bar

c3ec846

Change-Id: I5788448d580b53898e75fba68ff5d5a9d12e33d6

product-auto-label bot added size: s Pull request size is small. and removed size: xs Pull request size is extra small. labels Sep 23, 2022

tqdm.notebook

f50bf8b

Change-Id: I87e45085b7535083327a5fe2e51dba4b6411db00

aribray added the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 23, 2022

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 23, 2022

gcf-owl-bot bot and others added 5 commits September 23, 2022 17:03

🦉 Updates from OwlBot post-processor

0c41c9e

See https://github.com/googleapis/repo-automation-bots/blob/main/packages/owl-bot/README.md

reinclude ipywidgets

d679134

Change-Id: Ibe0fc01db05fcfaacdbe0c074b841ead3a39afc9

Merge branch 'aribray--tqdm' of github.com:aribray/python-bigquery in…

87212a5

…to aribray--tqdm Change-Id: I6448d5fb1d6a4e91a405b93b099ac00d748087d8

reinclude ipywidgets

be991b7

Change-Id: I56f8f98853e83ead0e0ca743c03407a521370233

change test assertions to tqdm_notebook

c6317a4

Change-Id: I2d55e529142ad0024ef4a98c2f15d10a73535380

product-auto-label bot added size: m Pull request size is medium. and removed size: s Pull request size is small. labels Sep 23, 2022

chalmerlowe reviewed Sep 23, 2022

View reviewed changes

google/cloud/bigquery/magics/magics.py Outdated Show resolved Hide resolved

aribray added 3 commits September 23, 2022 12:54

change test assertions in test_magics

13331b9

Change-Id: I7961ff1c5e9c54930d077e67ef9e01d79e351c5f

remove ipywidgets

d86ae58

Change-Id: I183e277fc7be8797c85d6802f4f8c3947871d4cc

update assertions in test

a6c3e88

Change-Id: I3b4a1b9460227ca49bf344362efbcc2c895d804d

aribray added the owlbot:run Add this label to trigger the Owlbot post processor. label Sep 23, 2022

aribray added 16 commits October 6, 2022 20:55

change mock patch to tqdm

99e108b

move assertion

3f53ad5

move assertions

4e901d4

add timeout side effect

30a3688

adjust import statement, mock.patch tqdm

122c5a5

create fixture

bf4b1c2

revert import change

84a5073

add import from helper

fd6b101

fix linting

9bedca6

Merge branch 'main' of github.com:aribray/python-bigquery into aribra…

a7e361c

…y--tqdm

remove unused imort

0dfc48d

set ipywidgets version to 7.7.1

ef3fc98

Merge branch 'main' of github.com:aribray/python-bigquery into aribra…

846e66a

…y--tqdm

set ipywidgets version to 7.7.1

05423ff

set ipywidgets version to 7.7.1

f06e44d

Merge branch 'main' of github.com:aribray/python-bigquery into aribra…

beb9675

…y--tqdm

aribray added the owlbot:run Add this label to trigger the Owlbot post processor. label Oct 10, 2022

gcf-owl-bot bot removed the owlbot:run Add this label to trigger the Owlbot post processor. label Oct 10, 2022

aribray added 2 commits October 10, 2022 17:41

bump sphinx version

f84f75c

bump sphinx version

b0c40c5

aribray requested a review from leahecole October 11, 2022 16:03

shollyman approved these changes Oct 11, 2022

View reviewed changes

google/cloud/bigquery/_tqdm_helpers.py Show resolved Hide resolved

aribray merged commit 506f781 into googleapis:main Oct 11, 2022

release-please bot mentioned this pull request Oct 11, 2022

chore(main): release 3.3.6 #1375

Merged

tswast reviewed Oct 12, 2022

View reviewed changes

chalmerlowe reviewed Oct 12, 2022

View reviewed changes

This was referenced Nov 8, 2022

[BQ] November 07, 2022 kitta65/bq-extension-vscode#115

Closed

[BQ] November 07, 2022 kitta65/prettier-plugin-bq#120

Closed

[BQ] November 07, 2022 kitta65/bq2cst#128

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: reconfigure tqdm progress bar in %%bigquery magic #1355

feat: reconfigure tqdm progress bar in %%bigquery magic #1355

aribray commented Sep 15, 2022 •

edited

Loading

shollyman left a comment

tswast left a comment

tswast Oct 12, 2022

chalmerlowe left a comment

chalmerlowe Oct 12, 2022

chalmerlowe Oct 12, 2022

feat: reconfigure tqdm progress bar in %%bigquery magic #1355

feat: reconfigure tqdm progress bar in %%bigquery magic #1355

Conversation

aribray commented Sep 15, 2022 • edited Loading

shollyman left a comment

Choose a reason for hiding this comment

tswast left a comment

Choose a reason for hiding this comment

tswast Oct 12, 2022

Choose a reason for hiding this comment

chalmerlowe left a comment

Choose a reason for hiding this comment

chalmerlowe Oct 12, 2022

Choose a reason for hiding this comment

chalmerlowe Oct 12, 2022

Choose a reason for hiding this comment

aribray commented Sep 15, 2022 •

edited

Loading