Add support for BigQuery External Data Tables #1466

rickard-von-essen · 2019-03-03T21:18:23Z

This adds support for BigQuery external data tables in Terraform.

This needs:

resourceBigQueryTableRead
Testing/verification
Documentation

[terraform] BQ - Support for External Data Tables

*bigquery*: add support for external data tables to `google_bigtable_instance`

googlebot · 2019-03-03T21:18:26Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here (e.g. I signed it!) and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

modular-magician · 2019-03-03T21:20:42Z

Hello! I am a robot who works on Magic Modules PRs.

I have detected that you are a community contributor, so your PR will be assigned to someone with a commit-bit on this repo for initial review. They will authorize it to run through our CI pipeline, which will generate downstream PRs.

Thanks for your contribution! A human will be with you soon.

rickard-von-essen · 2019-03-04T07:07:06Z

I signed it!

googlebot · 2019-03-04T07:07:09Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

rileykarson

Made a quick first pass, I have a few initial questions/comments

rileykarson · 2019-03-04T22:55:55Z

third_party/terraform/resources/resource_bigquery_table.go.erb

+						// BigtableOptions: [Optional] Additional options if sourceFormat is set to BIGTABLE.
+                                                "bigtable_options": {
+<% if version.nil? || version == 'ga' -%>
+							Removed: "This field is in beta. Use it in the the google-beta provider instead. See https://terraform.io/docs/providers/google/provider_versions.html for more details.",


is this in beta? It looks like it's just part of the v2 API like everything else.

externalDataConfiguration.sourceFormat | string | [Required] The data format. For CSV files, specify "CSV". For Google sheets, specify "GOOGLE_SHEETS". For newline-delimited JSON, specify "NEWLINE_DELIMITED_JSON". For Avro files, specify "AVRO". For Google Cloud Datastore backups, specify "DATASTORE_BACKUP". [Beta] For Google Cloud Bigtable, specify "BIGTABLE".

https://cloud.google.com/bigquery/docs/reference/rest/v2/tables

So maybe this should be included? But source_format disallow BIGTABLE unless using google-beta provider (as it already does). What do you think?

Ah, nope- you're correct. Let's omit bigtable_options entirely from the google provider, nested removed fields don't work well: https://github.com/hashicorp/terraform/issues/20756

third_party/terraform/resources/resource_bigquery_table.go.erb

rickard-von-essen · 2019-03-18T10:01:06Z

This is ready for review. I have done manual testing of most of this, but might add some test the coming days.

rickard-von-essen · 2019-03-29T07:19:45Z

Rebased

allanlegalstart · 2019-05-08T13:08:14Z

What is the status of this PR , it seems like it's currently in conflict ?

I can't wait to be able to define external tables with terraform, currently it's the only part of of our data pipeline that we have still manually defined

rileykarson · 2019-05-08T21:27:41Z

@allanlegalstart - ah, I managed to lose track of this unfortunately.

@rickard-von-essen - Really sorry for losing track of this! Do you mind rebasing again so I can make another pass? Alternatively, I can take over on the PR for you and get this merged if you've no longer got the time.

allanlegalstart · 2019-05-09T07:51:33Z

@rileykarson would be awesome if this can get merged in a near future :)

rickard-von-essen · 2019-05-09T09:12:56Z

One sec.

rickard-von-essen · 2019-05-09T09:19:09Z

Rebased

modular-magician · 2019-05-09T17:18:27Z

Hi! I'm the modular magician, I work on Magic Modules.
This PR seems not to have generated downstream PRs before, as of 91f2eed6a6b2ca6b81864fbb63a2197ca4eee9f3.

Pull request statuses

No diff detected in terraform-google-conversion.
No diff detected in Ansible.
No diff detected in Inspec.

New Pull Requests

I built this PR into one or more new PRs on other repositories, and when those are closed, this PR will also be merged and closed.
depends: hashicorp/terraform-provider-google-beta#696
depends: hashicorp/terraform-provider-google#3602

…s PR

And fixed wronly named option in docs s/schema_format/source_format/

…rom '"' (default) to ''

Tracked submodules are build/terraform-beta build/terraform-mapper build/terraform build/ansible build/inspec.

alfredo-gimenez · 2020-04-21T00:15:33Z

Hi, I noticed BigTable support was added then removed (6165666) and I don't see it in the lastest Terraform. Is there any planned support for external bigtable tables?

rileykarson · 2020-04-21T17:05:35Z

Not that I'm aware of- I believe the feature was disabled or otherwise untestable at the time of this PR. Can you file an issue asking for support against the terraform-provider-google repo?

rickard-von-essen · 2020-04-21T18:11:48Z

@alfredo-gimenez It was part of this PR for a short while (my goal was to support all external table sources) but I decided to remove the BigTable part since it was a huge part of the implementation, I had no enough time to test it properly, and I didn't have much knowledge of BigTable.

alfredo-gimenez · 2020-04-25T19:05:51Z

Gotcha, I’ll file an issue, thanks!

googlebot added the cla: no label Mar 3, 2019

modular-magician added the community label Mar 3, 2019

googlebot added cla: yes and removed cla: no labels Mar 4, 2019

rickard-von-essen force-pushed the bq-external-table branch from 004401b to cf1e8a4 Compare March 4, 2019 12:28

rileykarson self-requested a review March 4, 2019 15:25

rileykarson self-assigned this Mar 4, 2019

rickard-von-essen mentioned this pull request Mar 4, 2019

Request: Support creating BigQuery table definitions. hashicorp/terraform-provider-google#1055

Closed

rileykarson reviewed Mar 4, 2019

View reviewed changes

rickard-von-essen force-pushed the bq-external-table branch from 8e7d99e to 54cc6c4 Compare March 6, 2019 10:36

rickard-von-essen changed the title ~~WIP: BQ - Support for External Data Tables~~ BQ - Support for External Data Tables Mar 18, 2019

rickard-von-essen force-pushed the bq-external-table branch from 74a8673 to fe7f47b Compare March 18, 2019 09:33

rickard-von-essen force-pushed the bq-external-table branch from fe7f47b to df85fb3 Compare March 18, 2019 19:37

rickard-von-essen force-pushed the bq-external-table branch from df85fb3 to 3ca38e9 Compare March 29, 2019 07:19

rickard-von-essen force-pushed the bq-external-table branch from 3ca38e9 to 91f2eed Compare May 9, 2019 09:15

modular-magician added downstream-generated terraform-provider-google-beta terraform-provider-google labels May 9, 2019

rickard-von-essen and others added 21 commits July 3, 2019 20:18

Working with autodetect true

f31ea8c

google_sheets_options, max_bad_records, and ignore_unknown_values

a413226

csv_options

a55e9c7

Added bigtable_options

e037b93

Clearified Google Sheets range

cce28b8

Added EDC Schema

e753d5e

Updated docs for bigtable_options and some richer examples

cc20ad5

Added some links in the docs describing the JSON configs

5c883ce

Removed support for external table data from Bittable (beta) from thi…

6165666

…s PR

Fixed bug in schema handeling

8032cfc

Set some default values

8aa94d9

Fixed indentation

ada1e5a

Fixed return type of flattenExternalDataConfiguration

511dc86

Moved testBucketName function

fb59101

bigquery_table: Added External Data Configuration test

b63759f

Improved docs for quote

06e29db

BQ Ext Tables: removed schema, use top level instead

129bc2e

And fixed wronly named option in docs s/schema_format/source_format/

BQ Ext Tables: Properly handle nilable quote option in csv_options.

14ccee6

Re-arrange and put required before optional

124603b

Change a test to expose the problem with updating cvs_options.quote f…

3aec662

…rom '"' (default) to ''

Update tracked submodules -> HEAD on Wed Jul 3 20:18:17 UTC 2019

ed5b784

Tracked submodules are build/terraform-beta build/terraform-mapper build/terraform build/ansible build/inspec.

modular-magician force-pushed the bq-external-table branch from 7c178e3 to ed5b784 Compare July 3, 2019 20:18

modular-magician merged commit 9298c8d into GoogleCloudPlatform:master Jul 3, 2019

rileykarson mentioned this pull request Jul 8, 2019

Fix the quote value in BigQuery table external data config #2020

Merged

emilymye added the changelog: enhancement label Jul 12, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for BigQuery External Data Tables #1466

Add support for BigQuery External Data Tables #1466

rickard-von-essen commented Mar 3, 2019 •

edited by emilymye

Loading

googlebot commented Mar 3, 2019

modular-magician commented Mar 3, 2019

rickard-von-essen commented Mar 4, 2019

googlebot commented Mar 4, 2019

rileykarson left a comment

rileykarson Mar 4, 2019

rickard-von-essen Mar 5, 2019

rickard-von-essen Mar 5, 2019

rileykarson May 10, 2019

rickard-von-essen commented Mar 18, 2019

rickard-von-essen commented Mar 29, 2019

allanlegalstart commented May 8, 2019

rileykarson commented May 8, 2019 •

edited

Loading

allanlegalstart commented May 9, 2019

rickard-von-essen commented May 9, 2019

rickard-von-essen commented May 9, 2019

modular-magician commented May 9, 2019

alfredo-gimenez commented Apr 21, 2020

rileykarson commented Apr 21, 2020

rickard-von-essen commented Apr 21, 2020

alfredo-gimenez commented Apr 25, 2020

Add support for BigQuery External Data Tables #1466

Add support for BigQuery External Data Tables #1466

Conversation

rickard-von-essen commented Mar 3, 2019 • edited by emilymye Loading

[terraform] BQ - Support for External Data Tables

googlebot commented Mar 3, 2019

What to do if you already signed the CLA

Individual signers

Corporate signers

modular-magician commented Mar 3, 2019

rickard-von-essen commented Mar 4, 2019

googlebot commented Mar 4, 2019

rileykarson left a comment

Choose a reason for hiding this comment

rileykarson Mar 4, 2019

Choose a reason for hiding this comment

rickard-von-essen Mar 5, 2019

Choose a reason for hiding this comment

rickard-von-essen Mar 5, 2019

Choose a reason for hiding this comment

rileykarson May 10, 2019

Choose a reason for hiding this comment

rickard-von-essen commented Mar 18, 2019

rickard-von-essen commented Mar 29, 2019

allanlegalstart commented May 8, 2019

rileykarson commented May 8, 2019 • edited Loading

allanlegalstart commented May 9, 2019

rickard-von-essen commented May 9, 2019

rickard-von-essen commented May 9, 2019

modular-magician commented May 9, 2019

Pull request statuses

New Pull Requests

alfredo-gimenez commented Apr 21, 2020

rileykarson commented Apr 21, 2020

rickard-von-essen commented Apr 21, 2020

alfredo-gimenez commented Apr 25, 2020

rickard-von-essen commented Mar 3, 2019 •

edited by emilymye

Loading

rileykarson commented May 8, 2019 •

edited

Loading