Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add materialized views for bigquery #1726

Merged
merged 16 commits into from
Oct 4, 2023
Merged

Add materialized views for bigquery #1726

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
d1c48ee
add field and partition filter to time_partitioning for bq table
devuonocar Oct 3, 2023
282ed91
add materialized view
devuonocar Oct 3, 2023
91d8471
set optional fields for table
devuonocar Oct 4, 2023
9b211a3
Fix README.md
devuonocar Oct 4, 2023
3a00564
Update README.md
devuonocar Oct 4, 2023
97b8af8
Fix README.md
devuonocar Oct 4, 2023
405184d
Replace explicit dependencies with implicit ones
devuonocar Oct 4, 2023
f540fac
Fix indexes
devuonocar Oct 4, 2023
8fc510f
Merge branch 'master' into devuonocar/add-features-to-bigquery-table
ludoo Oct 4, 2023
d7197e1
Update table_id(s) values
devuonocar Oct 4, 2023
678aa1a
Try to fix tests
devuonocar Oct 4, 2023
54a33e3
Update tests
devuonocar Oct 4, 2023
0ade56f
Restore explicit dependencies
devuonocar Oct 4, 2023
1f21b99
Update README.md
devuonocar Oct 4, 2023
55570ca
Update README.md
devuonocar Oct 4, 2023
3b5c294
Merge branch 'master' into devuonocar/add-features-to-bigquery-table
ludoo Oct 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions modules/bigquery-dataset/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -294,9 +294,10 @@ module "bigquery-dataset" {
| [iam](variables.tf#L92) | IAM bindings in {ROLE => [MEMBERS]} format. Mutually exclusive with the access_* variables used for basic roles. | <code>map&#40;list&#40;string&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [labels](variables.tf#L103) | Dataset labels. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [location](variables.tf#L109) | Dataset location. | <code>string</code> | | <code>&#34;EU&#34;</code> |
| [materialized_views](variables.tf#L178) | Materialized views definitions. | <code title="map&#40;object&#40;&#123;&#10; query &#61; string&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; enable_refresh &#61; optional&#40;bool&#41;&#10; refresh_interval_ms &#61; optional&#40;bool&#41;&#10; allow_non_incremental_definition &#61; optional&#40;bool&#41;&#10; options &#61; optional&#40;object&#40;&#123;&#10; clustering &#61; optional&#40;list&#40;string&#41;&#41;&#10; expiration_time &#61; optional&#40;number&#41;&#10; &#125;&#41;, &#123;&#125;&#41;&#10; partitioning &#61; optional&#40;object&#40;&#123;&#10; field &#61; optional&#40;string&#41;&#10; range &#61; optional&#40;object&#40;&#123;&#10; end &#61; number&#10; interval &#61; number&#10; start &#61; number&#10; &#125;&#41;&#41;&#10; time &#61; optional&#40;object&#40;&#123;&#10; expiration_ms &#61; optional&#40;number&#41;&#10; field &#61; optional&#40;string&#41;&#10; type &#61; string&#10; require_partition_filter &#61; optional&#40;bool&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [options](variables.tf#L115) | Dataset options. | <code title="object&#40;&#123;&#10; default_collation &#61; optional&#40;string&#41;&#10; default_table_expiration_ms &#61; optional&#40;number&#41;&#10; default_partition_expiration_ms &#61; optional&#40;number&#41;&#10; delete_contents_on_destroy &#61; optional&#40;bool, false&#41;&#10; is_case_insensitive &#61; optional&#40;bool&#41;&#10; max_time_travel_hours &#61; optional&#40;number, 168&#41;&#10; storage_billing_model &#61; optional&#40;string&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>&#123;&#125;</code> |
| [tables](variables.tf#L134) | Table definitions. Options and partitioning default to null. Partitioning can only use `range` or `time`, set the unused one to null. | <code title="map&#40;object&#40;&#123;&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; schema &#61; optional&#40;string&#41;&#10; options &#61; optional&#40;object&#40;&#123;&#10; clustering &#61; optional&#40;list&#40;string&#41;&#41;&#10; encryption_key &#61; optional&#40;string&#41;&#10; expiration_time &#61; optional&#40;number&#41;&#10; &#125;&#41;, &#123;&#125;&#41;&#10; partitioning &#61; optional&#40;object&#40;&#123;&#10; field &#61; optional&#40;string&#41;&#10; range &#61; optional&#40;object&#40;&#123;&#10; end &#61; number&#10; interval &#61; number&#10; start &#61; number&#10; &#125;&#41;&#41;&#10; time &#61; optional&#40;object&#40;&#123;&#10; expiration_ms &#61; number&#10; type &#61; string&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [views](variables.tf#L163) | View definitions. | <code title="map&#40;object&#40;&#123;&#10; query &#61; string&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; use_legacy_sql &#61; optional&#40;bool&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [tables](variables.tf#L134) | Table definitions. Options and partitioning default to null. Partitioning can only use `range` or `time`, set the unused one to null. | <code title="map&#40;object&#40;&#123;&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; schema &#61; optional&#40;string&#41;&#10; options &#61; optional&#40;object&#40;&#123;&#10; clustering &#61; optional&#40;list&#40;string&#41;&#41;&#10; encryption_key &#61; optional&#40;string&#41;&#10; expiration_time &#61; optional&#40;number&#41;&#10; &#125;&#41;, &#123;&#125;&#41;&#10; partitioning &#61; optional&#40;object&#40;&#123;&#10; field &#61; optional&#40;string&#41;&#10; range &#61; optional&#40;object&#40;&#123;&#10; end &#61; number&#10; interval &#61; number&#10; start &#61; number&#10; &#125;&#41;&#41;&#10; time &#61; optional&#40;object&#40;&#123;&#10; expiration_ms &#61; optional&#40;number&#41;&#10; field &#61; optional&#40;string&#41;&#10; type &#61; string&#10; require_partition_filter &#61; optional&#40;bool&#41;&#10; &#125;&#41;&#41;&#10; &#125;&#41;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [views](variables.tf#L165) | View definitions. | <code title="map&#40;object&#40;&#123;&#10; query &#61; string&#10; deletion_protection &#61; optional&#40;bool&#41;&#10; description &#61; optional&#40;string, &#34;Terraform managed.&#34;&#41;&#10; friendly_name &#61; optional&#40;string&#41;&#10; labels &#61; optional&#40;map&#40;string&#41;, &#123;&#125;&#41;&#10; use_legacy_sql &#61; optional&#40;bool&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |

## Outputs

Expand Down
50 changes: 47 additions & 3 deletions modules/bigquery-dataset/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,9 +244,10 @@ resource "google_bigquery_table" "default" {
dynamic "time_partitioning" {
for_each = try(each.value.partitioning.time, null) != null ? [""] : []
content {
expiration_ms = each.value.partitioning.time.expiration_ms
field = each.value.partitioning.field
type = each.value.partitioning.time.type
expiration_ms = each.value.partitioning.time.expiration_ms
field = each.value.partitioning.time.field
type = each.value.partitioning.time.type
require_partition_filter = each.value.partitioning.time.require_partition_filter
}
}
}
Expand All @@ -267,3 +268,46 @@ resource "google_bigquery_table" "views" {
use_legacy_sql = each.value.use_legacy_sql
}
}

resource "google_bigquery_table" "materialized_view" {
depends_on = [google_bigquery_table.default]
for_each = var.materialized_views
project = var.project_id
dataset_id = google_bigquery_dataset.default.dataset_id
table_id = each.key
devuonocar marked this conversation as resolved.
Show resolved Hide resolved
friendly_name = each.value.friendly_name
description = each.value.description
labels = each.value.labels
clustering = each.value.options.clustering
expiration_time = each.value.options.expiration_time
deletion_protection = each.value.deletion_protection

dynamic "range_partitioning" {
for_each = try(each.value.partitioning.range, null) != null ? [""] : []
content {
field = each.value.partitioning.field
range {
start = each.value.partitioning.range.start
end = each.value.partitioning.range.end
interval = each.value.partitioning.range.interval
}
}
}

dynamic "time_partitioning" {
for_each = try(each.value.partitioning.time, null) != null ? [""] : []
content {
expiration_ms = each.value.partitioning.time.expiration_ms
field = each.value.partitioning.time.field
type = each.value.partitioning.time.type
require_partition_filter = each.value.partitioning.time.require_partition_filter
}
}

materialized_view {
query = each.value.query
enable_refresh = each.value.enable_refresh
refresh_interval_ms = each.value.refresh_interval_ms
allow_non_incremental_definition = each.value.allow_non_incremental_definition
}
}
39 changes: 37 additions & 2 deletions modules/bigquery-dataset/variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -152,8 +152,10 @@ variable "tables" {
start = number
}))
time = optional(object({
expiration_ms = number
type = string
expiration_ms = optional(number)
field = optional(string)
type = string
devuonocar marked this conversation as resolved.
Show resolved Hide resolved
require_partition_filter = optional(bool)
}))
}))
}))
Expand All @@ -172,3 +174,36 @@ variable "views" {
}))
default = {}
}

variable "materialized_views" {
description = "Materialized views definitions."
type = map(object({
query = string
deletion_protection = optional(bool)
description = optional(string, "Terraform managed.")
friendly_name = optional(string)
labels = optional(map(string), {})
enable_refresh = optional(bool)
refresh_interval_ms = optional(bool)
allow_non_incremental_definition = optional(bool)
options = optional(object({
clustering = optional(list(string))
expiration_time = optional(number)
}), {})
partitioning = optional(object({
field = optional(string)
range = optional(object({
end = number
interval = number
start = number
}))
time = optional(object({
expiration_ms = optional(number)
field = optional(string)
type = string
require_partition_filter = optional(bool)
}))
}))
}))
default = {}
}