Skip to content

Commit

Permalink
Merge pull request #1672 from luigi-bitonti/luigi-bitonti/pubsub-gcs-…
Browse files Browse the repository at this point in the history 
…feature

Added possibility to use gcs push endpoint on pubsub subscription
  • Loading branch information
@juliocc
juliocc authored Sep 13, 2023
2 parents e5d31b8 + 66416ac commit e19f332
Show file tree
Hide file tree
Showing 3 changed files with 78 additions and 15 deletions.
56 changes: 41 additions & 15 deletions modules/pubsub/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -124,6 +124,33 @@ module "pubsub" {
# tftest modules=1 resources=2
```

### Cloud Storage subscriptions

Cloud Storage subscriptions need extra configuration in the `cloud_storage_subscription_configs` variable.

```hcl
module "pubsub" {
source = "./fabric/modules/pubsub"
project_id = "my-project"
name = "my-topic"
subscriptions = {
test-cloudstorage = null
}
cloud_storage_subscription_configs = {
test-cloudstorage = {
bucket = "my-bucket"
filename_prefix = "test_prefix"
filename_suffix = "test_suffix"
max_duration = "100s"
max_bytes = 1000
avro_config = {
write_metadata = true
}
}
}
}
# tftest modules=1 resources=2
```
### Subscriptions with IAM

```hcl
Expand All @@ -144,25 +171,25 @@ module "pubsub" {
# tftest modules=1 resources=3
```
<!-- BEGIN TFDOC -->

## Variables

| name | description | type | required | default |
|---|---|:---:|:---:|:---:|
| [name](variables.tf#L79) | PubSub topic name. | <code>string</code> | ✓ | |
| [project_id](variables.tf#L84) | Project used for resources. | <code>string</code> | ✓ | |
| [name](variables.tf#L94) | PubSub topic name. | <code>string</code> | ✓ | |
| [project_id](variables.tf#L99) | Project used for resources. | <code>string</code> | ✓ | |
| [bigquery_subscription_configs](variables.tf#L17) | Configuration parameters for BigQuery subscriptions. | <code title="map&#40;object&#40;&#123;&#10; table &#61; string&#10; use_topic_schema &#61; bool&#10; write_metadata &#61; bool&#10; drop_unknown_fields &#61; bool&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [dead_letter_configs](variables.tf#L28) | Per-subscription dead letter policy configuration. | <code title="map&#40;object&#40;&#123;&#10; topic &#61; string&#10; max_delivery_attempts &#61; number&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [defaults](variables.tf#L37) | Subscription defaults for options. | <code title="object&#40;&#123;&#10; ack_deadline_seconds &#61; number&#10; message_retention_duration &#61; string&#10; retain_acked_messages &#61; bool&#10; expiration_policy_ttl &#61; string&#10; filter &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; ack_deadline_seconds &#61; null&#10; message_retention_duration &#61; null&#10; retain_acked_messages &#61; null&#10; expiration_policy_ttl &#61; null&#10; filter &#61; null&#10;&#125;">&#123;&#8230;&#125;</code> |
| [iam](variables.tf#L55) | IAM bindings for topic in {ROLE => [MEMBERS]} format. | <code>map&#40;list&#40;string&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [kms_key](variables.tf#L61) | KMS customer managed encryption key. | <code>string</code> | | <code>null</code> |
| [labels](variables.tf#L67) | Labels. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [message_retention_duration](variables.tf#L73) | Minimum duration to retain a message after it is published to the topic. | <code>string</code> | | <code>null</code> |
| [push_configs](variables.tf#L89) | Push subscription configurations. | <code title="map&#40;object&#40;&#123;&#10; attributes &#61; map&#40;string&#41;&#10; endpoint &#61; string&#10; oidc_token &#61; object&#40;&#123;&#10; audience &#61; string&#10; service_account_email &#61; string&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [regions](variables.tf#L102) | List of regions used to set persistence policy. | <code>list&#40;string&#41;</code> | | <code>&#91;&#93;</code> |
| [schema](variables.tf#L108) | Topic schema. If set, all messages in this topic should follow this schema. | <code title="object&#40;&#123;&#10; definition &#61; string&#10; msg_encoding &#61; optional&#40;string, &#34;ENCODING_UNSPECIFIED&#34;&#41;&#10; schema_type &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [subscription_iam](variables.tf#L118) | IAM bindings for subscriptions in {SUBSCRIPTION => {ROLE => [MEMBERS]}} format. | <code>map&#40;map&#40;list&#40;string&#41;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [subscriptions](variables.tf#L124) | Topic subscriptions. Also define push configs for push subscriptions. If options is set to null subscription defaults will be used. Labels default to topic labels if set to null. | <code title="map&#40;object&#40;&#123;&#10; labels &#61; map&#40;string&#41;&#10; options &#61; object&#40;&#123;&#10; ack_deadline_seconds &#61; number&#10; message_retention_duration &#61; string&#10; retain_acked_messages &#61; bool&#10; expiration_policy_ttl &#61; string&#10; filter &#61; string&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [cloud_storage_subscription_configs](variables.tf#L28) | Configuration parameters for Cloud Storage subscriptions. | <code title="map&#40;object&#40;&#123;&#10; bucket &#61; string&#10; filename_prefix &#61; string&#10; filename_suffix &#61; string&#10; max_duration &#61; string&#10; max_bytes &#61; number&#10; avro_config &#61; object&#40;&#123;&#10; write_metadata &#61; bool&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [dead_letter_configs](variables.tf#L43) | Per-subscription dead letter policy configuration. | <code title="map&#40;object&#40;&#123;&#10; topic &#61; string&#10; max_delivery_attempts &#61; number&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [defaults](variables.tf#L52) | Subscription defaults for options. | <code title="object&#40;&#123;&#10; ack_deadline_seconds &#61; number&#10; message_retention_duration &#61; string&#10; retain_acked_messages &#61; bool&#10; expiration_policy_ttl &#61; string&#10; filter &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code title="&#123;&#10; ack_deadline_seconds &#61; null&#10; message_retention_duration &#61; null&#10; retain_acked_messages &#61; null&#10; expiration_policy_ttl &#61; null&#10; filter &#61; null&#10;&#125;">&#123;&#8230;&#125;</code> |
| [iam](variables.tf#L70) | IAM bindings for topic in {ROLE => [MEMBERS]} format. | <code>map&#40;list&#40;string&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [kms_key](variables.tf#L76) | KMS customer managed encryption key. | <code>string</code> | | <code>null</code> |
| [labels](variables.tf#L82) | Labels. | <code>map&#40;string&#41;</code> | | <code>&#123;&#125;</code> |
| [message_retention_duration](variables.tf#L88) | Minimum duration to retain a message after it is published to the topic. | <code>string</code> | | <code>null</code> |
| [push_configs](variables.tf#L104) | Push subscription configurations. | <code title="map&#40;object&#40;&#123;&#10; attributes &#61; map&#40;string&#41;&#10; endpoint &#61; string&#10; oidc_token &#61; object&#40;&#123;&#10; audience &#61; string&#10; service_account_email &#61; string&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [regions](variables.tf#L117) | List of regions used to set persistence policy. | <code>list&#40;string&#41;</code> | | <code>&#91;&#93;</code> |
| [schema](variables.tf#L123) | Topic schema. If set, all messages in this topic should follow this schema. | <code title="object&#40;&#123;&#10; definition &#61; string&#10; msg_encoding &#61; optional&#40;string, &#34;ENCODING_UNSPECIFIED&#34;&#41;&#10; schema_type &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> | | <code>null</code> |
| [subscription_iam](variables.tf#L133) | IAM bindings for subscriptions in {SUBSCRIPTION => {ROLE => [MEMBERS]}} format. | <code>map&#40;map&#40;list&#40;string&#41;&#41;&#41;</code> | | <code>&#123;&#125;</code> |
| [subscriptions](variables.tf#L139) | Topic subscriptions. Also define push configs for push subscriptions. If options is set to null subscription defaults will be used. Labels default to topic labels if set to null. | <code title="map&#40;object&#40;&#123;&#10; labels &#61; map&#40;string&#41;&#10; options &#61; object&#40;&#123;&#10; ack_deadline_seconds &#61; number&#10; message_retention_duration &#61; string&#10; retain_acked_messages &#61; bool&#10; expiration_policy_ttl &#61; string&#10; filter &#61; string&#10; &#125;&#41;&#10;&#125;&#41;&#41;">map&#40;object&#40;&#123;&#8230;&#125;&#41;&#41;</code> | | <code>&#123;&#125;</code> |

## Outputs

Expand All @@ -174,5 +201,4 @@ module "pubsub" {
| [subscription_id](outputs.tf#L36) | Subscription ids. | |
| [subscriptions](outputs.tf#L46) | Subscription resources. | |
| [topic](outputs.tf#L54) | Topic resource. | |

<!-- END TFDOC -->
22 changes: 22 additions & 0 deletions modules/pubsub/main.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ locals {
}
]
])
metadata_config = {
for k, v in var.cloud_storage_subscription_configs : k => v.avro_config
}
oidc_config = {
for k, v in var.push_configs : k => v.oidc_token
}
Expand Down Expand Up @@ -127,6 +130,25 @@ resource "google_pubsub_subscription" "default" {
drop_unknown_fields = var.bigquery_subscription_configs[each.key].drop_unknown_fields
}
}

dynamic "cloud_storage_config" {
for_each = try(var.cloud_storage_subscription_configs[each.key], null) == null ? [] : [""]
content {
bucket = var.cloud_storage_subscription_configs[each.key].bucket
filename_prefix = var.cloud_storage_subscription_configs[each.key].filename_prefix
filename_suffix = var.cloud_storage_subscription_configs[each.key].filename_suffix
max_duration = var.cloud_storage_subscription_configs[each.key].max_duration
max_bytes = var.cloud_storage_subscription_configs[each.key].max_bytes
dynamic "avro_config" {
for_each = (
local.metadata_config[each.key] == null ? [] : [""]
)
content {
write_metadata = local.metadata_config[each.key].write_metadata
}
}
}
}
}

resource "google_pubsub_subscription_iam_binding" "default" {
Expand Down
15 changes: 15 additions & 0 deletions modules/pubsub/variables.tf
View file
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,21 @@ variable "bigquery_subscription_configs" {
default = {}
}

variable "cloud_storage_subscription_configs" {
description = "Configuration parameters for Cloud Storage subscriptions."
type = map(object({
bucket = string
filename_prefix = string
filename_suffix = string
max_duration = string
max_bytes = number
avro_config = object({
write_metadata = bool
})
}))
default = {}
}

variable "dead_letter_configs" {
description = "Per-subscription dead letter policy configuration."
type = map(object({
Expand Down

0 comments on commit e19f332

Please sign in to comment.