Skip to content

Latest commit

 

History

History
194 lines (169 loc) · 10.2 KB

README.md

File metadata and controls

194 lines (169 loc) · 10.2 KB
Raw

Google Cloud Pub/Sub Module

This module allows managing a single Pub/Sub topic, including multiple subscriptions and IAM bindings at the topic and subscriptions levels, as well as schemas.

Examples

Simple topic with IAM

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  iam = {
    "roles/pubsub.viewer"     = ["group:${var.group_email}"]
    "roles/pubsub.subscriber" = ["serviceAccount:${var.service_account.email}"]
  }
}
# tftest modules=1 resources=3  inventory=simple.yaml e2e

Topic with schema

module "topic_with_schema" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  schema = {
    msg_encoding = "JSON"
    schema_type  = "AVRO"
    definition = jsonencode({
      "type" = "record",
      "name" = "Avro",
      "fields" = [{
        "name" = "StringField",
        "type" = "string"
        },
        {
          "name" = "FloatField",
          "type" = "float"
        },
        {
          "name" = "BooleanField",
          "type" = "boolean"
        },
      ]
    })
  }
}
# tftest modules=1 resources=2 inventory=schema.yaml e2e

Subscriptions

Subscriptions are defined with the subscriptions variable, allowing optional configuration of per-subscription defaults. Push subscriptions need extra configuration, shown in the following example.

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  labels     = { test = "default" }
  subscriptions = {
    test-pull = {}
    test-pull-override = {
      labels                = { test = "override" }
      retain_acked_messages = true
    }
  }
}
# tftest modules=1 resources=3 inventory=subscriptions.yaml e2e

Push subscriptions

Push subscriptions need extra configuration in the push_configs variable.

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  subscriptions = {
    test-push = {
      push = {
        endpoint = "https://example.com/foo"
      }
    }
  }
}
# tftest modules=1 resources=2 inventory=push-subscription.yaml e2e

BigQuery subscriptions

BigQuery subscriptions need extra configuration in the bigquery_subscription_configs variable.

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  subscriptions = {
    test-bigquery = {
      bigquery = {
        table               = "${module.bigquery-dataset.tables["my_table"].project}:${module.bigquery-dataset.tables["my_table"].dataset_id}.${module.bigquery-dataset.tables["my_table"].table_id}"
        use_topic_schema    = true
        write_metadata      = false
        drop_unknown_fields = true
      }
    }
  }
}
# tftest modules=2 resources=5 fixtures=fixtures/bigquery-dataset.tf inventory=bigquery-subscription.yaml e2e

Cloud Storage subscriptions

Cloud Storage subscriptions need extra configuration in the cloud_storage_subscription_configs variable.

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  subscriptions = {
    test-cloudstorage = {
      cloud_storage = {
        bucket          = module.gcs.name
        filename_prefix = var.prefix
        filename_suffix = "test_suffix"
        max_duration    = "100s"
        max_bytes       = 1000
        avro_config = {
          write_metadata = true
        }
      }
    }
  }
}
# tftest modules=2 resources=4 fixtures=fixtures/gcs.tf inventory=cloud-storage-subscription.yaml e2e

Subscriptions with IAM

module "pubsub" {
  source     = "./fabric/modules/pubsub"
  project_id = var.project_id
  name       = "my-topic"
  subscriptions = {
    test-1 = {
      iam = {
        "roles/pubsub.subscriber" = ["serviceAccount:${var.service_account.email}"]
      }
    }
  }
}
# tftest modules=1 resources=3 inventory=subscription-iam.yaml e2e

Variables

name description type required default
name PubSub topic name. string ✓
project_id Project used for resources. string ✓
iam IAM bindings for topic in {ROLE => [MEMBERS]} format. map(list(string)) {}
iam_bindings Authoritative IAM bindings in {KEY => {role = ROLE, members = [], condition = {}}}. Keys are arbitrary. map(object({…})) {}
iam_bindings_additive Keyring individual additive IAM bindings. Keys are arbitrary. map(object({…})) {}
kms_key KMS customer managed encryption key. string null
labels Labels. map(string) {}
message_retention_duration Minimum duration to retain a message after it is published to the topic. string null
regions List of regions used to set persistence policy. list(string) []
schema Topic schema. If set, all messages in this topic should follow this schema. object({…}) null
subscriptions Topic subscriptions. Also define push configs for push subscriptions. If options is set to null subscription defaults will be used. Labels default to topic labels if set to null. map(object({…})) {}

Outputs

name description sensitive
id Fully qualified topic id.
schema Schema resource.
schema_id Schema resource id.
subscription_id Subscription ids.
subscriptions Subscription resources.
topic Topic resource.

Fixtures