Skip to content

Commit

Permalink
Add materialized views for bigquery (GoogleCloudPlatform#1726)
Browse files Browse the repository at this point in the history 
* add field and partition filter to time_partitioning for bq table

* add materialized view

* set optional fields for table

* Fix README.md

* Update README.md

* Fix README.md

* Replace explicit dependencies with implicit ones

* Fix indexes

* Update table_id(s) values

* Try to fix tests

* Update tests

* Restore explicit dependencies

* Update README.md

* Update README.md

---------

Co-authored-by: Ludovico Magnocavallo <[email protected]>
  • Loading branch information
@devuonocar @ludoo
devuonocar and ludoo authored Oct 4, 2023
1 parent 256d690 commit 7503bce
Show file tree
Hide file tree
Showing 3 changed files with 89 additions and 9 deletions.
9 changes: 5 additions & 4 deletions modules/bigquery-dataset/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,7 @@ module "bigquery-dataset" {
| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [id](variables.tf#L98) | Dataset id. | <code>string</code> | ✓ | |
| [project_id](variables.tf#L129) | Id of the project where datasets will be created. | <code>string</code> | ✓ | |
| [project_id](variables.tf#L162) | Id of the project where datasets will be created. | <code>string</code> | ✓ | |
| [access](variables.tf#L17) | Map of access rules with role and identity type. Keys are arbitrary and must match those in the `access_identities` variable, types are `domain`, `group`, `special_group`, `user`, `view`. | <code title="map&#40;object&#40;&#123;&#10; role &#61; string&#10; type &#61; string&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [access_identities](variables.tf#L33) | Map of access identities used for basic access roles. View identities have the format 'project_id\|dataset_id\|table_id'. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [authorized_datasets](variables.tf#L39) | An array of datasets to be authorized on the dataset. | <code title="list&#40;object&#40;&#123;&#10; dataset_id &#61; string,&#10; project_id &#61; string,&#10;&#125;&#41;&#41;">list&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#91;&#93;</code> |
Expand All @@ -294,9 +294,10 @@ module "bigquery-dataset" {
| [iam](variables.tf#L92) | IAM bindings in {ROLE => [MEMBERS]} format. Mutually exclusive with the access_* variables used for basic roles. | <code>map&#40;list&#40;string&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [labels](variables.tf#L103) | Dataset labels. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [location](variables.tf#L109) | Dataset location. | <code>string</code> | | <code>&#34;EU&#34;</code> |
| [options](variables.tf#L115) | Dataset options. | <code title="object&#40;&#123;&#10; default_collation &#61; optional&#40;string&#41;&#10; default_table_expiration_ms &#61; optional&#40;number&#41;&#10; default_partition_expiration_ms &#61; optional&#40;number&#41;&#10; delete_contents_on_destroy &#61; optional&#40;bool, false&#41;&#10; is_case_insensitive &#61; optional&#40;bool&#41;&#10; max_time_travel_hours &#61; optional&#40;number, 168&#41;&#10; storage_billing_model &#61; optional&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>&#123;&#125;</code> |
| [tables](variables.tf#L134) | Table definitions. Options and partitioning default to null. Partitioning can only use `range` or `time`, set the unused one to null. | <code title="map&#40;object&#40;&#123;&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; schema &#61; optional&#40;string&#41;&#10; options &#61; optional&#40;object&#40;&#123;&#10; clustering &#61; optional&#40;list&#40;string&#41;&#41;&#10; encryption_key &#61; optional&#40;string&#41;&#10; expiration_time &#61; optional&#40;number&#41;&#10; &#125;&#41;, &#123;&#125;&#41;&#10; partitioning &#61; optional&#40;object&#40;&#123;&#10; field &#61; optional&#40;string&#41;&#10; range &#61; optional&#40;object&#40;&#123;&#10; end &#61; number&#10; interval &#61; number&#10; start &#61; number&#10; &#125;&#41;&#41;&#10; time &#61; optional&#40;object&#40;&#123;&#10; expiration_ms &#61; number&#10; type &#61; string&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [views](variables.tf#L163) | View definitions. | <code title="map&#40;object&#40;&#123;&#10; query &#61; string&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; use_legacy_sql &#61; optional&#40;bool&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [materialized_views](variables.tf#L115) | Materialized views definitions. | <code title="map&#40;object&#40;&#123;&#10; query &#61; string&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; enable_refresh &#61; optional&#40;bool&#41;&#10; refresh_interval_ms &#61; optional&#40;bool&#41;&#10; allow_non_incremental_definition &#61; optional&#40;bool&#41;&#10; options &#61; optional&#40;object&#40;&#123;&#10; clustering &#61; optional&#40;list&#40;string&#41;&#41;&#10; expiration_time &#61; optional&#40;number&#41;&#10; &#125;&#41;, &#123;&#125;&#41;&#10; partitioning &#61; optional&#40;object&#40;&#123;&#10; field &#61; optional&#40;string&#41;&#10; range &#61; optional&#40;object&#40;&#123;&#10; end &#61; number&#10; interval &#61; number&#10; start &#61; number&#10; &#125;&#41;&#41;&#10; time &#61; optional&#40;object&#40;&#123;&#10; type &#61; string&#10; expiration_ms &#61; optional&#40;number&#41;&#10; field &#61; optional&#40;string&#41;&#10; require_partition_filter &#61; optional&#40;bool&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [options](variables.tf#L148) | Dataset options. | <code title="object&#40;&#123;&#10; default_collation &#61; optional&#40;string&#41;&#10; default_table_expiration_ms &#61; optional&#40;number&#41;&#10; default_partition_expiration_ms &#61; optional&#40;number&#41;&#10; delete_contents_on_destroy &#61; optional&#40;bool, false&#41;&#10; is_case_insensitive &#61; optional&#40;bool&#41;&#10; max_time_travel_hours &#61; optional&#40;number, 168&#41;&#10; storage_billing_model &#61; optional&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>&#123;&#125;</code> |
| [tables](variables.tf#L167) | Table definitions. Options and partitioning default to null. Partitioning can only use `range` or `time`, set the unused one to null. | <code title="map&#40;object&#40;&#123;&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; schema &#61; optional&#40;string&#41;&#10; options &#61; optional&#40;object&#40;&#123;&#10; clustering &#61; optional&#40;list&#40;string&#41;&#41;&#10; encryption_key &#61; optional&#40;string&#41;&#10; expiration_time &#61; optional&#40;number&#41;&#10; &#125;&#41;, &#123;&#125;&#41;&#10; partitioning &#61; optional&#40;object&#40;&#123;&#10; field &#61; optional&#40;string&#41;&#10; range &#61; optional&#40;object&#40;&#123;&#10; end &#61; number&#10; interval &#61; number&#10; start &#61; number&#10; &#125;&#41;&#41;&#10; time &#61; optional&#40;object&#40;&#123;&#10; type &#61; string&#10; expiration_ms &#61; optional&#40;number&#41;&#10; field &#61; optional&#40;string&#41;&#10; require_partition_filter &#61; optional&#40;bool&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [views](variables.tf#L198) | View definitions. | <code title="map&#40;object&#40;&#123;&#10; query &#61; string&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; use_legacy_sql &#61; optional&#40;bool&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |

## Outputs

Expand Down
50 changes: 47 additions & 3 deletions modules/bigquery-dataset/main.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -244,9 +244,10 @@ resource "google_bigquery_table" "default" {
dynamic "time_partitioning" {
for_each = try(each.value.partitioning.time, null) != null ? [""] : []
content {
expiration_ms = each.value.partitioning.time.expiration_ms
field = each.value.partitioning.field
type = each.value.partitioning.time.type
expiration_ms = each.value.partitioning.time.expiration_ms
field = each.value.partitioning.time.field
type = each.value.partitioning.time.type
require_partition_filter = each.value.partitioning.time.require_partition_filter
}
}
}
Expand All @@ -267,3 +268,46 @@ resource "google_bigquery_table" "views" {
use_legacy_sql = each.value.use_legacy_sql
}
}

resource "google_bigquery_table" "materialized_view" {
depends_on = [google_bigquery_table.default]
for_each = var.materialized_views
project = var.project_id
dataset_id = google_bigquery_dataset.default.dataset_id
table_id = each.key
friendly_name = each.value.friendly_name
description = each.value.description
labels = each.value.labels
clustering = each.value.options.clustering
expiration_time = each.value.options.expiration_time
deletion_protection = each.value.deletion_protection

dynamic "range_partitioning" {
for_each = try(each.value.partitioning.range, null) != null ? [""] : []
content {
field = each.value.partitioning.field
range {
start = each.value.partitioning.range.start
end = each.value.partitioning.range.end
interval = each.value.partitioning.range.interval
}
}
}

dynamic "time_partitioning" {
for_each = try(each.value.partitioning.time, null) != null ? [""] : []
content {
expiration_ms = each.value.partitioning.time.expiration_ms
field = each.value.partitioning.time.field
type = each.value.partitioning.time.type
require_partition_filter = each.value.partitioning.time.require_partition_filter
}
}

materialized_view {
query = each.value.query
enable_refresh = each.value.enable_refresh
refresh_interval_ms = each.value.refresh_interval_ms
allow_non_incremental_definition = each.value.allow_non_incremental_definition
}
}
39 changes: 37 additions & 2 deletions modules/bigquery-dataset/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,39 @@ variable "location" {
default = "EU"
}

variable "materialized_views" {
description = "Materialized views definitions."
type = map(object({
query = string
deletion_protection = optional(bool)
description = optional(string, "Terraform managed.")
friendly_name = optional(string)
labels = optional(map(string), {})
enable_refresh = optional(bool)
refresh_interval_ms = optional(bool)
allow_non_incremental_definition = optional(bool)
options = optional(object({
clustering = optional(list(string))
expiration_time = optional(number)
}), {})
partitioning = optional(object({
field = optional(string)
range = optional(object({
end = number
interval = number
start = number
}))
time = optional(object({
type = string
expiration_ms = optional(number)
field = optional(string)
require_partition_filter = optional(bool)
}))
}))
}))
default = {}
}

variable "options" {
description = "Dataset options."
type = object({
Expand Down Expand Up @@ -152,8 +185,10 @@ variable "tables" {
start = number
}))
time = optional(object({
expiration_ms = number
type = string
type = string
expiration_ms = optional(number)
field = optional(string)
require_partition_filter = optional(bool)
}))
}))
}))
Expand Down

0 comments on commit 7503bce

Please sign in to comment.