Only load fields that are needed from Google API for Big Query schema.

[jezdez]

What type of PR is this? (check all applicable)

• Other

Description

While working with Big Query we noticed a pretty big amount of data being transferred from when fetching Big Query schema data. This change basically only loads the data that we really need for the schema.

Related Tickets & Documents

I used the Google API Explorer to build the values for the fields parameters: e.g. https://developers.google.com/apis-explorer/#p/bigquery/v2/bigquery.tables.get

More info for the fields parameter: https://developers.google.com/api-client-library/python/guide/performance#partial-response-fields-parameter

[arikfr]

Hi,

(This is a template message, but I mean every word of it. Also you're welcome to reply)

Thank you for making this contribution. While we couldn't bring it to completion and merge, it's still very much appreciated. 🙇

In the past year the Redash code base gone under massive updates: on the backend we moved to Python 3 & RQ instead of Celery and on the frontend we replaced Angular with React. It's very likely this makes this PR irrelevant without significant changes. :-(

I'm closing this PR now. But if you're still interested in making it happen, let me know and I will reopen.

Thanks.

[jezdez]

@arikfr I think this would still be useful since it reduces the amount of data requested from BQ quite a bit.

[arikfr]

@jezdez 👍 I wonder how to reliably test this?

[jezdez]

@arikfr No idea how to test this properly, no. I just updated the PR though since it fixes another issue with the BigQuery query runner of not using pagination for the dataset API.

[guidopetri]

@jezdez , thanks for the PR! We've updated a lot of things now that we're Community-driven so - if you're still interested in getting this merged - would you mind rebasing off master to re-run the CI, as well as updating merge conflicts?

We're trying to clean up our PR todo list, so if you're not interested, that's fine - we'll close the PR in about a week if we don't hear back. If you're interested in reopening the PR afterwards, we would also very much welcome that.

[justinclift]

@jezdez This does sound really useful. Hoping you're interested? 😄