Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add BigQuery subscriptions to Pubsub module. #997

Merged
merged 2 commits into from
Nov 21, 2022
Merged

Add BigQuery subscriptions to Pubsub module. #997

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
e692eac
Add BigQuery subcriptions to Pubsub module.
iht Nov 21, 2022
ce05246
Merge branch 'master' into bigquery_subscriptions
juliocc Nov 21, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 38 additions & 14 deletions modules/pubsub/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@

This module allows managing a single Pub/Sub topic, including multiple subscriptions and IAM bindings at the topic and subscriptions levels, as well as schemas.


## Examples

### Simple topic with IAM
Expand Down Expand Up @@ -101,6 +100,30 @@ module "pubsub" {
# tftest modules=1 resources=2
```

### BigQuery subscriptions

BigQuery subscriptions need extra configuration in the `bigquery_subscription_configs` variable.

```hcl
module "pubsub" {
source = "./fabric/modules/pubsub"
project_id = "my-project"
name = "my-topic"
subscriptions = {
test-bigquery = null
}
bigquery_subscription_configs = {
test-bigquery = {
table = "my_project_id:my_dataset.my_table"
use_topic_schema = true
write_metadata = false
drop_unknown_fields = true
}
}
}
# tftest modules=1 resources=2
```

### Subscriptions with IAM

```hcl
Expand All @@ -126,19 +149,20 @@ module "pubsub" {

| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [name](variables.tf#L68) | PubSub topic name. | <code>string</code> | ✓ | |
| [project_id](variables.tf#L73) | Project used for resources. | <code>string</code> | ✓ | |
| [dead_letter_configs](variables.tf#L17) | Per-subscription dead letter policy configuration. | <code title="map&#40;object&#40;&#123;&#10; topic &#61; string&#10; max_delivery_attempts &#61; number&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [defaults](variables.tf#L26) | Subscription defaults for options. | <code title="object&#40;&#123;&#10; ack_deadline_seconds &#61; number&#10; message_retention_duration &#61; string&#10; retain_acked_messages &#61; bool&#10; expiration_policy_ttl &#61; string&#10; filter &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; ack_deadline_seconds &#61; null&#10; message_retention_duration &#61; null&#10; retain_acked_messages &#61; null&#10; expiration_policy_ttl &#61; null&#10; filter &#61; null&#10;&#125;">&#123;&#8230;&#125;</code> |
| [iam](variables.tf#L44) | IAM bindings for topic in {ROLE => [MEMBERS]} format. | <code>map&#40;list&#40;string&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [kms_key](variables.tf#L50) | KMS customer managed encryption key. | <code>string</code> | | <code>null</code> |
| [labels](variables.tf#L56) | Labels. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [message_retention_duration](variables.tf#L62) | Minimum duration to retain a message after it is published to the topic. | <code>string</code> | | <code>null</code> |
| [push_configs](variables.tf#L78) | Push subscription configurations. | <code title="map&#40;object&#40;&#123;&#10; attributes &#61; map&#40;string&#41;&#10; endpoint &#61; string&#10; oidc_token &#61; object&#40;&#123;&#10; audience &#61; string&#10; service_account_email &#61; string&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [regions](variables.tf#L91) | List of regions used to set persistence policy. | <code>list&#40;string&#41;</code> | | <code>&#91;&#93;</code> |
| [schema](variables.tf#L97) | Topic schema. If set, all messages in this topic should follow this schema. | <code title="object&#40;&#123;&#10; definition &#61; string&#10; msg_encoding &#61; optional&#40;string, &#34;ENCODING_UNSPECIFIED&#34;&#41;&#10; schema_type &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [subscription_iam](variables.tf#L107) | IAM bindings for subscriptions in {SUBSCRIPTION => {ROLE => [MEMBERS]}} format. | <code>map&#40;map&#40;list&#40;string&#41;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [subscriptions](variables.tf#L113) | Topic subscriptions. Also define push configs for push subscriptions. If options is set to null subscription defaults will be used. Labels default to topic labels if set to null. | <code title="map&#40;object&#40;&#123;&#10; labels &#61; map&#40;string&#41;&#10; options &#61; object&#40;&#123;&#10; ack_deadline_seconds &#61; number&#10; message_retention_duration &#61; string&#10; retain_acked_messages &#61; bool&#10; expiration_policy_ttl &#61; string&#10; filter &#61; string&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [name](variables.tf#L79) | PubSub topic name. | <code>string</code> | ✓ | |
| [project_id](variables.tf#L84) | Project used for resources. | <code>string</code> | ✓ | |
| [bigquery_subscription_configs](variables.tf#L17) | Configuration parameters for BigQuery subscriptions. | <code title="map&#40;object&#40;&#123;&#10; table &#61; string&#10; use_topic_schema &#61; bool&#10; write_metadata &#61; bool&#10; drop_unknown_fields &#61; bool&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [dead_letter_configs](variables.tf#L28) | Per-subscription dead letter policy configuration. | <code title="map&#40;object&#40;&#123;&#10; topic &#61; string&#10; max_delivery_attempts &#61; number&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [defaults](variables.tf#L37) | Subscription defaults for options. | <code title="object&#40;&#123;&#10; ack_deadline_seconds &#61; number&#10; message_retention_duration &#61; string&#10; retain_acked_messages &#61; bool&#10; expiration_policy_ttl &#61; string&#10; filter &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; ack_deadline_seconds &#61; null&#10; message_retention_duration &#61; null&#10; retain_acked_messages &#61; null&#10; expiration_policy_ttl &#61; null&#10; filter &#61; null&#10;&#125;">&#123;&#8230;&#125;</code> |
| [iam](variables.tf#L55) | IAM bindings for topic in {ROLE => [MEMBERS]} format. | <code>map&#40;list&#40;string&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [kms_key](variables.tf#L61) | KMS customer managed encryption key. | <code>string</code> | | <code>null</code> |
| [labels](variables.tf#L67) | Labels. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [message_retention_duration](variables.tf#L73) | Minimum duration to retain a message after it is published to the topic. | <code>string</code> | | <code>null</code> |
| [push_configs](variables.tf#L89) | Push subscription configurations. | <code title="map&#40;object&#40;&#123;&#10; attributes &#61; map&#40;string&#41;&#10; endpoint &#61; string&#10; oidc_token &#61; object&#40;&#123;&#10; audience &#61; string&#10; service_account_email &#61; string&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [regions](variables.tf#L102) | List of regions used to set persistence policy. | <code>list&#40;string&#41;</code> | | <code>&#91;&#93;</code> |
| [schema](variables.tf#L108) | Topic schema. If set, all messages in this topic should follow this schema. | <code title="object&#40;&#123;&#10; definition &#61; string&#10; msg_encoding &#61; optional&#40;string, &#34;ENCODING_UNSPECIFIED&#34;&#41;&#10; schema_type &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [subscription_iam](variables.tf#L118) | IAM bindings for subscriptions in {SUBSCRIPTION => {ROLE => [MEMBERS]}} format. | <code>map&#40;map&#40;list&#40;string&#41;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [subscriptions](variables.tf#L124) | Topic subscriptions. Also define push configs for push subscriptions. If options is set to null subscription defaults will be used. Labels default to topic labels if set to null. | <code title="map&#40;object&#40;&#123;&#10; labels &#61; map&#40;string&#41;&#10; options &#61; object&#40;&#123;&#10; ack_deadline_seconds &#61; number&#10; message_retention_duration &#61; string&#10; retain_acked_messages &#61; bool&#10; expiration_policy_ttl &#61; string&#10; filter &#61; string&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |

## Outputs

Expand Down
10 changes: 10 additions & 0 deletions modules/pubsub/main.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,16 @@ resource "google_pubsub_subscription" "default" {
}
}
}

dynamic "bigquery_config" {
for_each = try(var.bigquery_subscription_configs[each.key], null) == null ? [] : [""]
content {
table = var.bigquery_subscription_configs[each.key].table
use_topic_schema = var.bigquery_subscription_configs[each.key].use_topic_schema
write_metadata = var.bigquery_subscription_configs[each.key].write_metadata
drop_unknown_fields = var.bigquery_subscription_configs[each.key].drop_unknown_fields
}
}
}

resource "google_pubsub_subscription_iam_binding" "default" {
Expand Down
11 changes: 11 additions & 0 deletions modules/pubsub/variables.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,17 @@
* limitations under the License.
*/

variable "bigquery_subscription_configs" {
description = "Configuration parameters for BigQuery subscriptions."
type = map(object({
table = string
use_topic_schema = bool
write_metadata = bool
drop_unknown_fields = bool
}))
default = {}
}

variable "dead_letter_configs" {
description = "Per-subscription dead letter policy configuration."
type = map(object({
Expand Down