docs: add samples from bigquery_storage/to_dataframe #50

plamut · 2020-09-02T13:36:18Z

Samples from https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/bigquery_storage/to_dataframe

IMPORTANT: When merging, use the rebase and merge option to preserve the samples commit history!

Closes #51.

…ogleCloudPlatform/python-docs-samples#1994) * BigQuery Storage API sample for reading pandas dataframe How to get a pandas DataFrame, fast! The first two examples use the existing BigQuery client. These examples create a thread pool and read in parallel. The final example shows using just the new BigQuery Storage client, but only shows how to read with a single thread.

…gleCloudPlatform/python-docs-samples#2088) * Remove temporary dataset from bqstorage pandas tutorial As of google-cloud-bigquery version 1.11.1, the `to_dataframe` method will fallback to the tabledata.list API when the BigQuery Storage API fails to read the query results. * Remove unused imports

…oogleCloudPlatform/python-docs-samples#2087) * Add magics tutorial with BigQuery Storage API integration. This is a notebooks tutorial, modeled after the Jupyter notebook example code for BigQuery. Use some caution when running these tests, as they run some large-ish (5 GB processed) queries and download about 500 MB worth of data. This is intentional, as the BigQuery Storage API is most useful for downloading large results. * Update deps. * Don't run big queries on Travis.

…oogleCloudPlatform/python-docs-samples#2269)

…oogleCloudPlatform/python-docs-samples#2413)

…ocs-samples#2436) * Adds updates including compute * Python 2 compat pytest * Fixing weird \r\n issue from GH merge * Put asset tests back in * Re-add pod operator test * Hack parameter for k8s pod operator

…amples#2005) * Auto-update dependencies. * Revert update of appengine/flexible/datastore. * revert update of appengine/flexible/scipy * revert update of bigquery/bqml * revert update of bigquery/cloud-client * revert update of bigquery/datalab-migration * revert update of bigtable/quickstart * revert update of compute/api * revert update of container_registry/container_analysis * revert update of dataflow/run_template * revert update of datastore/cloud-ndb * revert update of dialogflow/cloud-client * revert update of dlp * revert update of functions/imagemagick * revert update of functions/ocr/app * revert update of healthcare/api-client/fhir * revert update of iam/api-client * revert update of iot/api-client/gcs_file_to_device * revert update of iot/api-client/mqtt_example * revert update of language/automl * revert update of run/image-processing * revert update of vision/automl * revert update testing/requirements.txt * revert update of vision/cloud-client/detect * revert update of vision/cloud-client/product_search * revert update of jobs/v2/api_client * revert update of jobs/v3/api_client * revert update of opencensus * revert update of translate/cloud-client * revert update to speech/cloud-client Co-authored-by: Kurtis Van Gent <[email protected]> Co-authored-by: Doug Mahugh <[email protected]>

…eCloudPlatform/python-docs-samples#2724) Co-authored-by: Leah E. Cole <[email protected]>

…0 [(#3050)](GoogleCloudPlatform/python-docs-samples#3050) * chore(deps): update dependency google-cloud-bigquery-storage to v0.8.0 * chore(deps): update pandas-gbq * chore(deps): update ipython * chore: update requirements.txt * chore: it is spelled version. * chore(deps): split pandas version * chore(deps): split pandas version Co-authored-by: Christopher Wilcox <[email protected]> Co-authored-by: Leah Cole <[email protected]>

…ples#2806) * chore(deps): update dependency requests to v2.23.0 * Simplify noxfile and add version control. * Configure appengine/standard to only test Python 2.7. * Update Kokokro configs to match noxfile. * Add requirements-test to each folder. * Remove Py2 versions from everything execept appengine/standard. * Remove conftest.py. * Remove appengine/standard/conftest.py * Remove 'no-sucess-flaky-report' from pytest.ini. * Add GAE SDK back to appengine/standard tests. * Fix typo. * Roll pytest to python 2 version. * Add a bunch of testing requirements. * Remove typo. * Add appengine lib directory back in. * Add some additional requirements. * Fix issue with flake8 args. * Even more requirements. * Readd appengine conftest.py. * Add a few more requirements. * Even more Appengine requirements. * Add webtest for appengine/standard/mailgun. * Add some additional requirements. * Add workaround for issue with mailjet-rest. * Add responses for appengine/standard/mailjet. Co-authored-by: Renovate Bot <[email protected]>

…m/python-docs-samples#3148) Co-authored-by: Leah E. Cole <[email protected]>

…49)](GoogleCloudPlatform/python-docs-samples#3049) * chore(deps): update dependency google-cloud-bigquery to v1.24.0 * chore(deps): update ipython version * fix: fix requirements order * explicitly add grpc to resolve errors * adjust arguments * undo mistake * bump auth version Co-authored-by: Bu Sun Kim <[email protected]> Co-authored-by: Leah Cole <[email protected]> Co-authored-by: Leah E. Cole <[email protected]> Co-authored-by: Christopher Wilcox <[email protected]>

GoogleCloudPlatform/python-docs-samples#2797) * chore(deps): update dependency google-auth-oauthlib to v0.4.1 * resolve dependency finding errors * fix new matplotlib error Co-authored-by: Leah E. Cole <[email protected]> Co-authored-by: Leah Cole <[email protected]> Co-authored-by: Bu Sun Kim <[email protected]> Co-authored-by: Christopher Wilcox <[email protected]>

…thon-docs-samples#3188) Co-authored-by: Leah E. Cole <[email protected]>

…eCloudPlatform/python-docs-samples#3464) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | patch | `==1.14.0` -> `==1.14.1` | | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | minor | `==1.11.2` -> `==1.14.1` | --- ### Release Notes <details> <summary>googleapis/google-auth-library-python</summary> ### [`v1.14.1`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#1141-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1140v1141-2020-04-21) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.14.0...v1.14.1) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples).

…eCloudPlatform/python-docs-samples#3724) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | patch | `==1.14.1` -> `==1.14.2` | --- ### Release Notes <details> <summary>googleapis/google-auth-library-python</summary> ### [`v1.14.2`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#1142-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1141v1142-2020-05-07) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.14.1...v1.14.2) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples).

…eCloudPlatform/python-docs-samples#3728) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [google-auth](https://togithub.com/googleapis/google-auth-library-python) | patch | `==1.14.2` -> `==1.14.3` | --- ### Release Notes <details> <summary>googleapis/google-auth-library-python</summary> ### [`v1.14.3`](https://togithub.com/googleapis/google-auth-library-python/blob/master/CHANGELOG.md#1143-httpswwwgithubcomgoogleapisgoogle-auth-library-pythoncomparev1142v1143-2020-05-11) [Compare Source](https://togithub.com/googleapis/google-auth-library-python/compare/v1.14.2...v1.14.3) </details> --- ### Renovate configuration :date: **Schedule**: At any time (no schedule defined). :vertical_traffic_light: **Automerge**: Disabled by config. Please merge this manually once you are satisfied. :recycle: **Rebasing**: Never, or you tick the rebase/retry checkbox. :no_bell: **Ignore**: Close this PR and you won't be reminded about this update again. --- - [x] If you want to rebase/retry this PR, check this box --- This PR has been generated by [WhiteSource Renovate](https://renovate.whitesourcesoftware.com). View repository job log [here](https://app.renovatebot.com/dashboard#GoogleCloudPlatform/python-docs-samples).

…udPlatform/python-docs-samples#3806)

…hon-docs-samples#3815)

…eCloudPlatform/python-docs-samples#3903)

…3962)

…eCloudPlatform/python-docs-samples#3968) Co-authored-by: Leah E. Cole <[email protected]>

…m/python-docs-samples#4058)

…eCloudPlatform/python-docs-samples#4073)

…m/python-docs-samples#4083)

…m/python-docs-samples#4125)

…Platform/python-docs-samples#4279) * chore(deps): update dependency pytest to v5.4.3 * specify pytest for python 2 in appengine Co-authored-by: Leah Cole <[email protected]>

…eCloudPlatform/python-docs-samples#4293)

google-cla · 2020-09-02T13:36:55Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

google-cla · 2020-09-02T14:05:34Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

plamut · 2020-09-02T14:52:49Z

The tests fail, because they expect the results to contain Python objects, but the actual results are Pyarrow objects, e.g. 123 vs. <pyarrow.Int64Scalar: 123>.

@shollyman Is this actually expected behavior and pyarrow values need to be converted to Python objects explicitly, or should ReadRowsStream.rows() convert the results automatically before handing them to the user?

Update:
It's the former. Taking one of the BigQuery DB API tests as an example, mocked results data actually contains pyarrow values (created with the _to_pyarrow() test helper), implying that bqstorage indeed returns pyarrow values, and user code needs to account for that.

busunkim96 · 2020-09-02T15:24:28Z

It looks like the regular Kokoro job is also running the sample tests https://github.com/googleapis/python-bigquery-storage/blob/master/noxfile.py#L135

There seems to only be one existing sample in this repo. Here is what I suggest:

move the quickstart into dataframe or some other directory nested under samples
re-run synthtool
delete the samples nox session in the library's main noxfile.py
merge PR
follow with a docs update internally

google-cla · 2020-09-03T12:38:03Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

google-cla · 2020-09-03T13:04:11Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

google-cla · 2020-09-03T14:32:02Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

plamut · 2020-09-03T14:33:59Z

@busunkim96 Done. I had to tweak a few things (e.g. dependencies), but now the samples tests pass locally - both the migrated to_dataframe/ and the existing quickstart which was moved to its own subdirectory.

google-cla · 2020-09-03T14:40:15Z

All (the pull request submitter and all commit authors) CLAs are signed, but one or more commits were authored or co-authored by someone other than the pull request submitter.

We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that by leaving a comment that contains only @googlebot I consent. in this pull request.

Note to project maintainer: There may be cases where the author cannot leave a comment, or the comment is not properly detected as consent. In those cases, you can manually confirm consent of the commit author(s), and set the cla label to yes (if enabled on your project).

ℹ️ Googlers: Go here for more info.

plamut · 2020-09-10T17:21:33Z

@busunkim96 Please also appease the CLA bot on my behalf, thanks!

tswast and others added 30 commits February 7, 2019 10:53

Add bigquerystorage_jupyter_tutorial_query_default region tag.

ce20d43

BigQuery Storage: Update to use faster Arrow data format. [(#2269)](G…

db2b426

…oogleCloudPlatform/python-docs-samples#2269)

Update pandas/bqstorage samples to latest library changes. [(#2413)](G…

0b94250

…oogleCloudPlatform/python-docs-samples#2413)

Adds updates including compute [(#2436)](GoogleCloudPlatform/python-d…

3ac2b1a

…ocs-samples#2436) * Adds updates including compute * Python 2 compat pytest * Fixing weird \r\n issue from GH merge * Put asset tests back in * Re-add pod operator test * Hack parameter for k8s pod operator

chore(deps): update dependency google-auth to v1.11.2 [(#2724)](Googl…

4bfc841

…eCloudPlatform/python-docs-samples#2724) Co-authored-by: Leah E. Cole <[email protected]>

Update dependency google-auth to v1.14.0 [(#3148)](GoogleCloudPlatfor…

f87f1ba

…m/python-docs-samples#3148) Co-authored-by: Leah E. Cole <[email protected]>

Update dependency pyarrow to v0.17.0 [(#3188)](GoogleCloudPlatform/py…

520a38a

…thon-docs-samples#3188) Co-authored-by: Leah E. Cole <[email protected]>

chore(deps): update dependency pyarrow to v0.17.1 [(#3806)](GoogleClo…

b4e0be6

…udPlatform/python-docs-samples#3806)

update google-auth to 1.15.0 part 2 [(#3815)](GoogleCloudPlatform/pyt…

8481d86

…hon-docs-samples#3815)

chore(deps): update dependency google-auth to v1.16.0 [(#3903)](Googl…

4daaff5

…eCloudPlatform/python-docs-samples#3903)

update google-auth [(#3962)](GoogleCloudPlatform/python-docs-samples#…

3da2869

…3962)

Update dependency google-cloud-bigquery-storage to v1 [(#3968)](Googl…

16b3b56

…eCloudPlatform/python-docs-samples#3968) Co-authored-by: Leah E. Cole <[email protected]>

Update dependency google-auth to v1.17.0 [(#4058)](GoogleCloudPlatfor…

e9628e3

…m/python-docs-samples#4058)

chore(deps): update dependency google-auth to v1.17.1 [(#4073)](Googl…

20cde69

…eCloudPlatform/python-docs-samples#4073)

Update dependency google-auth to v1.17.2 [(#4083)](GoogleCloudPlatfor…

5a96253

…m/python-docs-samples#4083)

Update dependency google-auth to v1.18.0 [(#4125)](GoogleCloudPlatfor…

ebaa1f8

…m/python-docs-samples#4125)

chore(deps): update dependency pytest to v5.4.3 [(#4279)](GoogleCloud…

1f1e160

…Platform/python-docs-samples#4279) * chore(deps): update dependency pytest to v5.4.3 * specify pytest for python 2 in appengine Co-authored-by: Leah Cole <[email protected]>

chore(deps): update dependency google-auth to v1.19.0 [(#4293)](Googl…

5069bee

…eCloudPlatform/python-docs-samples#4293)

Merge branch 'to_dataframe-samples-2' into add-to_dataframe-samples-2

9fc5355

google-cla bot added the cla: no This human has *not* signed the Contributor License Agreement. label Sep 2, 2020

chore: update templates

b4deb53

plamut requested review from busunkim96 and shollyman September 2, 2020 14:09

product-auto-label bot added the api: bigquerystorage Issues related to the googleapis/python-bigquery-storage API. label Sep 3, 2020

Adjust test assertions to pyarrow values

5407056

plamut force-pushed the add-to_dataframe-samples-2 branch from 8df5a86 to 5407056 Compare September 3, 2020 13:03

Remove samples session from the main noxfile

a1cc972

Move quickstart sample into its own subdirectory

8c2fe1a

plamut force-pushed the add-to_dataframe-samples-2 branch from f891b9a to 8c2fe1a Compare September 3, 2020 14:39

plamut requested review from busunkim96 and removed request for busunkim96 September 10, 2020 10:38

busunkim96 approved these changes Sep 10, 2020

View reviewed changes

busunkim96 added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Sep 10, 2020

plamut merged commit ba84d5b into googleapis:master Sep 10, 2020

plamut deleted the add-to_dataframe-samples-2 branch September 10, 2020 17:25

This was referenced Sep 11, 2020

Remove BigQuery Storage samples (moved to library repo) GoogleCloudPlatform/python-docs-samples#4666

Closed

chore: remove BigQuery Storage to_dataframe samples (moved) GoogleCloudPlatform/python-docs-samples#4667

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

docs: add samples from bigquery_storage/to_dataframe #50

docs: add samples from bigquery_storage/to_dataframe #50

plamut commented Sep 2, 2020 •

edited

Loading

google-cla bot commented Sep 2, 2020

google-cla bot commented Sep 2, 2020

plamut commented Sep 2, 2020 •

edited

Loading

busunkim96 commented Sep 2, 2020

google-cla bot commented Sep 3, 2020

google-cla bot commented Sep 3, 2020

google-cla bot commented Sep 3, 2020

plamut commented Sep 3, 2020

google-cla bot commented Sep 3, 2020

plamut commented Sep 10, 2020

docs: add samples from bigquery_storage/to_dataframe #50

docs: add samples from bigquery_storage/to_dataframe #50

Conversation

plamut commented Sep 2, 2020 • edited Loading

google-cla bot commented Sep 2, 2020

google-cla bot commented Sep 2, 2020

plamut commented Sep 2, 2020 • edited Loading

busunkim96 commented Sep 2, 2020

google-cla bot commented Sep 3, 2020

google-cla bot commented Sep 3, 2020

google-cla bot commented Sep 3, 2020

plamut commented Sep 3, 2020

google-cla bot commented Sep 3, 2020

plamut commented Sep 10, 2020

plamut commented Sep 2, 2020 •

edited

Loading

plamut commented Sep 2, 2020 •

edited

Loading