Add support for range partitioning in BigQuery

[tristan957]

Currently only time partitioning is supported.

Release Note Template for Downstream PRs (will be copied)

hashicorp/terraform-provider-google#5239

Support for range-based partitioning in BigQuery tables has landed. Fixes https://github.com/terraform-providers/terraform-provider-google/issues/5239.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

[tristan957]

@googlebot I signed it!

[googlebot]

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

[tristan957]

This is going to need support from maintainers to show me how to make the schema beta-only, since the documentation for range-based partitioning says it is in beta.

[modular-magician]

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review. They will authorize it to run through our CI pipeline, which will generate downstream PRs.

Thanks for your contribution! A human will be with you soon.

@emilymye, please review this PR or find an appropriate assignee.

[emilymye]

A few comments:

• I added just one comment example, but for our handwritte code, it's pretty much just ERB version checks around the beta-only code. However, time_partioning is also marked as experimental and isn't version checked.

• You'll need to set the range partioning values when Create/Update() and set the state value from the API in Read(). I'm going to be slightly lazy and just make you ctrl-f for "time_partitioning " rather than post links to each section, but they give pretty good examples of what the code should look like, especially the expand/flattenTimePartitioning helpers. As a note, we wrote this resource a while back, using the now-technically-deprecated autogenerated Golang library, but it still supports [RangePartitioning](https://godoc.org/google.golang.org/api/bigquery/v2#RangePartitioning]

• Will you be able to run tests by yourself? If not, let me know when you're ready and I can run them in a test env to verify.

[tristan957]

@emilymye I tried setting up the environment for magic modules, but either got stuck or stopped after installing the gems. I will have to go back to see the reason for it. Will update the PR later today.

[emilymye]

@tristan957 let me know if you have any issues and I can try to help!

[tristan957]

PR has been updated besides the indentation which I left a comment on

[tristan957]

@emilymye I do not have a Google Cloud account to test with unfortunately. I am no longer on the project that would benefit from this PR, but I told them I would see it to completion anyway. I have also added the code for mirroring the time partitioning code.

[tristan957]

I also did end up getting the build environment setup. If I had an account to test with, it looks like I would be able to do that as well.

[emilymye]

no worries! once you make the last few changes I can run the tests for you :)

[tristan957]

Shouldn't these version gating flags be unless version == 'beta'?

[tristan957]

I see that all of my added code is in terraform-provider-google even though it is a beta feature

[emilymye]

unless version == 'ga' means it will add that code unless the version is ga - are you seeing the code in GA when you compile locally? or do you mean in the downstream PR? I haven't rerun the CI tool yet so the downstream hasn't been updated yet

[tristan957]

That makes a lot of sense. Will fix the testing suite after work 👍 . I was probably just blind before.

[tristan957]

Testing has now been fixed

[emilymye]

@tristan957 sorry for the re-review process - I just realized a couple of things were missing. After that, it should be good and I'll start the generator again!

[tristan957]

@emilymye pushed fixes for latest review

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff
Terraform Beta: Diff

[tristan957]

Unsure of why the diffs report that output since I have not touched those lines.

[emilymye]

@tristan957 the diff is a new way of generating downstream changes that we might be transitioning to in the middle of this PR! Exciting! You shouldn't need to do anything but do you mind rebasing your branch? It looks like the diff are showing you reverting some changes that other PRs made.

[tristan957]

@emilymye rebased 👍

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff
Terraform Beta: Diff

[emilymye]

@tristan957 thank you so much for your patience! there's just a few things left but I ran the test with fixed syntax and it passed, so so once these last things are in, I'll go ahead and submit the PR! As you may have noticed, we recently started a new generation system that outputs the diff automatically in a comment, so it'll be much easier/faster for me from now on to check the diff and test your changes. It's so close!

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 2 insertions(+), 21 deletions(-))
Terraform Beta: Diff ( 3 files changed, 2 insertions(+), 192 deletions(-))

[tristan957]

@emilymye no problem. I am having trouble building locally, but hopefully all your changes were addressed correctly. All that needs to happen is a bundle compile and go compile, and hopefully all will be good to go.

I also went ahead and rebased into the one commit.

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 21 insertions(+), 2 deletions(-))
Terraform Beta: Diff ( 4 files changed, 208 insertions(+), 2 deletions(-))

[modular-magician]

Hi! I'm the modular magician. Your PR generated some diffs in downstreams - here they are.

Diff report:

Terraform GA: Diff ( 2 files changed, 21 insertions(+), 2 deletions(-))
Terraform Beta: Diff ( 3 files changed, 189 insertions(+), 2 deletions(-))

[tristan957]

@emilymye thanks for the approval. At what point could I see this in a beta provider release? Granted the code hasn't been merged yet.

[emilymye]

@tristan957 just merged, was confirming some tests! Thank you so much for contributing! I'm so sorry that it took so long to get this feature in and I missed the release cut for the release (3.10) next monday/tuesday, so it'll be in probably 3.11.0 or whatever is after 3.10 week after next. Hopefully you should have no trouble contributing in the future since you've now dealt with some of the hardest things to do!

[tristan957]

@emilymye thank you for your help in getting this merged. No problem with the length of time. Pretty sure 3 major holidays passed between when it opened and when it closed. Who wants to do work during the holidays? :) If I ever need another feature for the plugin, I know where to go.

[sid0911]

Team, I am getting following error while using the range_partitioning feature of bigquery. Any help ?

resource "google_bigquery_table" "bq_tables_rgpt" {
project = var.project_id
dataset_id = google_bigquery_dataset.tera_gcp_dtset1.dataset_id
table_id = "btd_ebim_fal_product_agg_mnth"
friendly_name = "Weather Loactions"
schema = file("${path.module}/files/createTableSqls/rangept/btd_ebim_fal_product_agg_mnth.json")
range_partitioning {
field = "year_month"
range {
start = 200001
end = 209912
interval = 1
}
}
depends_on = [google_bigquery_dataset.tera_gcp_dtset1]
}

---

Error: Unsupported block type

on bigqtable.tf line 51, in resource "google_bigquery_table" "bq_tables_rgpt":
51: range_partitioning {

Blocks of type "range_partitioning" are not expected here.

[tristan957]

@sid0911 are you using the google-beta provider