From 7e2a42a6ce30f520b59761195cee54210e1579a4 Mon Sep 17 00:00:00 2001 From: The Magician Date: Wed, 21 Aug 2019 12:16:16 -0700 Subject: [PATCH] Autogenerate BigQuery dataset (#180) Signed-off-by: Modular Magician --- docs/resources/google_bigquery_dataset.md | 16 ++++++++++------ docs/resources/google_bigquery_datasets.md | 2 ++ libraries/google_bigquery_dataset.rb | 4 ++++ libraries/google_bigquery_datasets.rb | 4 ++++ 4 files changed, 20 insertions(+), 6 deletions(-) diff --git a/docs/resources/google_bigquery_dataset.md b/docs/resources/google_bigquery_dataset.md index 6725df161..ef19eccd7 100644 --- a/docs/resources/google_bigquery_dataset.md +++ b/docs/resources/google_bigquery_dataset.md @@ -35,15 +35,15 @@ end ## Properties Properties that can be accessed from the `google_bigquery_dataset` resource: - * `access`: Access controls on the bucket. + * `access`: An array of objects that define dataset access for one or more entities. * `domain`: A domain to grant access to. Any users signed in with the domain specified will be granted the specified access - * `group_by_email`: An email address of a Google Group to grant access to + * `group_by_email`: An email address of a Google Group to grant access to. - * `role`: Describes the rights granted to the user specified by the other member of the access object + * `role`: Describes the rights granted to the user specified by the other member of the access object. Primitive, Predefined and custom roles are supported. Predefined roles that have equivalent primitive roles are swapped by the API to their Primitive counterparts, and will show a diff post-create. See [official docs](https://cloud.google.com/bigquery/docs/access-control). - * `special_group`: A special group to grant access to. + * `special_group`: A special group to grant access to. Possible values include: * `projectOwners`: Owners of the enclosing project. * `projectReaders`: Readers of the enclosing project. * `projectWriters`: Writers of the enclosing project. * `allAuthenticatedUsers`: All authenticated BigQuery users. * `user_by_email`: An email address of a user to grant access to. For example: fred@example.com @@ -57,10 +57,14 @@ Properties that can be accessed from the `google_bigquery_dataset` resource: * `project_id`: The ID of the project containing this dataset. - * `default_table_expiration_ms`: The default lifetime of all tables in the dataset, in milliseconds + * `default_table_expiration_ms`: The default lifetime of all tables in the dataset, in milliseconds. The minimum value is 3600000 milliseconds (one hour). Once this property is set, all newly-created tables in the dataset will have an `expirationTime` property set to the creation time plus the value in this property, and changing the value will only affect new tables, not existing ones. When the `expirationTime` for a given table is reached, that table will be deleted automatically. If a table's `expirationTime` is modified or removed before the table expires, or if you provide an explicit `expirationTime` when creating a table, that value takes precedence over the default expiration time indicated by this property. + + * `default_partition_expiration_ms`: The default partition expiration for all partitioned tables in the dataset, in milliseconds. Once this property is set, all newly-created partitioned tables in the dataset will have an `expirationMs` property in the `timePartitioning` settings set to this value, and changing the value will only affect new tables, not existing ones. The storage in a partition will have an expiration time of its partition time plus this value. Setting this property overrides the use of `defaultTableExpirationMs` for partitioned tables: only one of `defaultTableExpirationMs` and `defaultPartitionExpirationMs` will be used for any new partitioned table. If you provide an explicit `timePartitioning.expirationMs` when creating or updating a partitioned table, that value takes precedence over the default partition expiration time indicated by this property. * `description`: A user-friendly description of the dataset + * `etag`: A hash of the resource. + * `friendly_name`: A descriptive name for the dataset * `id`: The fully-qualified unique name of the dataset in the format projectId:datasetId. The dataset name without the project name is given in the datasetId field @@ -69,7 +73,7 @@ Properties that can be accessed from the `google_bigquery_dataset` resource: * `last_modified_time`: The date when this dataset or any of its tables was last modified, in milliseconds since the epoch. - * `location`: The geographic location where the dataset should reside. Possible values include EU and US. The default value is US. + * `location`: The geographic location where the dataset should reside. See [official docs](https://cloud.google.com/bigquery/docs/dataset-locations). There are two types of locations, regional or multi-regional. A regional location is a specific geographic place, such as Tokyo, and a multi-regional location is a large geographic area, such as the United States, that contains at least two geographic places. Possible regional values include: `asia-east1`, `asia-northeast1`, `asia-southeast1`, `australia-southeast1`, `europe-north1`, `europe-west2` and `us-east4`. Possible multi-regional values: `EU` and `US`. The default value is multi-regional location `US`. Changing this forces a new resource to be created. diff --git a/docs/resources/google_bigquery_datasets.md b/docs/resources/google_bigquery_datasets.md index 9d3e0b494..f36469a30 100644 --- a/docs/resources/google_bigquery_datasets.md +++ b/docs/resources/google_bigquery_datasets.md @@ -20,6 +20,8 @@ Properties that can be accessed from the `google_bigquery_datasets` resource: See [google_bigquery_dataset.md](google_bigquery_dataset.md) for more detailed information * `dataset_references`: an array of `google_bigquery_dataset` dataset_reference + * `default_partition_expiration_ms`: an array of `google_bigquery_dataset` default_partition_expiration_ms + * `etags`: an array of `google_bigquery_dataset` etag * `friendly_names`: an array of `google_bigquery_dataset` friendly_name * `ids`: an array of `google_bigquery_dataset` id * `labels`: an array of `google_bigquery_dataset` labels diff --git a/libraries/google_bigquery_dataset.rb b/libraries/google_bigquery_dataset.rb index a7816fe80..1d6386fe7 100644 --- a/libraries/google_bigquery_dataset.rb +++ b/libraries/google_bigquery_dataset.rb @@ -28,7 +28,9 @@ class Dataset < GcpResourceBase attr_reader :creation_time attr_reader :dataset_reference attr_reader :default_table_expiration_ms + attr_reader :default_partition_expiration_ms attr_reader :description + attr_reader :etag attr_reader :friendly_name attr_reader :id attr_reader :labels @@ -47,7 +49,9 @@ def parse @creation_time = @fetched['creationTime'] @dataset_reference = GoogleInSpec::BigQuery::Property::DatasetDatasetReference.new(@fetched['datasetReference'], to_s) @default_table_expiration_ms = @fetched['defaultTableExpirationMs'] + @default_partition_expiration_ms = @fetched['defaultPartitionExpirationMs'] @description = @fetched['description'] + @etag = @fetched['etag'] @friendly_name = @fetched['friendlyName'] @id = @fetched['id'] @labels = @fetched['labels'] diff --git a/libraries/google_bigquery_datasets.rb b/libraries/google_bigquery_datasets.rb index 2f791e7d9..60ae1ed8f 100644 --- a/libraries/google_bigquery_datasets.rb +++ b/libraries/google_bigquery_datasets.rb @@ -24,6 +24,8 @@ class Datasets < GcpResourceBase filter_table_config = FilterTable.create filter_table_config.add(:dataset_references, field: :dataset_reference) + filter_table_config.add(:default_partition_expiration_ms, field: :default_partition_expiration_ms) + filter_table_config.add(:etags, field: :etag) filter_table_config.add(:friendly_names, field: :friendly_name) filter_table_config.add(:ids, field: :id) filter_table_config.add(:labels, field: :labels) @@ -68,6 +70,8 @@ def transform(key, value) def transformers { 'datasetReference' => ->(obj) { return :dataset_reference, GoogleInSpec::BigQuery::Property::DatasetDatasetReference.new(obj['datasetReference'], to_s) }, + 'defaultPartitionExpirationMs' => ->(obj) { return :default_partition_expiration_ms, obj['defaultPartitionExpirationMs'] }, + 'etag' => ->(obj) { return :etag, obj['etag'] }, 'friendlyName' => ->(obj) { return :friendly_name, obj['friendlyName'] }, 'id' => ->(obj) { return :id, obj['id'] }, 'labels' => ->(obj) { return :labels, obj['labels'] },