cloud-dataplex

Cloud Dataplex instance with lake, zone & assests

This module manages the creation of Cloud Dataplex instance along with lake, zone & assets in single regions.

Simple example

This example shows how to setup a Cloud Dataplex instance, lake, zone & asset creation in GCP project.

module "dataplex" {
  source     = "./fabric/modules/cloud-dataplex"
  name       = "terraform-lake"
  prefix     = "test"
  project_id = "myproject"
  region     = "europe-west2"
  zones = {
    landing = {
      type      = "RAW"
      discovery = true
      assets = {
        gcs_1 = {
          resource_name          = "gcs_bucket"
          cron_schedule          = "15 15 * * *"
          discovery_spec_enabled = true
          resource_spec_type     = "STORAGE_BUCKET"
        }
      }
    },
    curated = {
      type      = "CURATED"
      discovery = false
      assets = {
        bq_1 = {
          resource_name          = "bq_dataset"
          cron_schedule          = null
          discovery_spec_enabled = false
          resource_spec_type     = "BIGQUERY_DATASET"
        }
      }
    }
  }
}

# tftest modules=1 resources=5

IAM

This example shows how to setup a Cloud Dataplex instance, lake, zone & asset creation in GCP project assigning IAM roles at lake and zone level.

module "dataplex" {
  source     = "./fabric/modules/cloud-dataplex"
  name       = "lake"
  prefix     = "test"
  project_id = "myproject"
  region     = "europe-west2"
  iam = {
    "roles/dataplex.viewer" = [
      "group:[email protected]",
      "group:[email protected]"
    ]
  }
  zones = {
    landing = {
      type      = "RAW"
      discovery = true
      assets = {
        gcs_1 = {
          resource_name          = "gcs_bucket"
          cron_schedule          = "15 15 * * *"
          discovery_spec_enabled = true
          resource_spec_type     = "STORAGE_BUCKET"
        }
      }
    },
    curated = {
      type      = "CURATED"
      discovery = false
      iam = {
        "roles/viewer" = [
          "group:[email protected]",
          "group:[email protected]"
        ]
        "roles/dataplex.dataReader" = [
          "group:[email protected]",
          "group:[email protected]"
        ]
      }
      assets = {
        bq_1 = {
          resource_name          = "bq_dataset"
          cron_schedule          = null
          discovery_spec_enabled = false
          resource_spec_type     = "BIGQUERY_DATASET"
        }
      }
    }
  }
}

# tftest modules=1 resources=8

TODO

• support multi-regions

Variables

name	description	type	required	default
name	Name of Dataplex Lake.	string	✓
project_id	The ID of the project where this Dataplex Lake will be created.	string	✓
region	Region of the Dataplax Lake.	string	✓
zones	Dataplex lake zones, such as RAW and CURATED.	map(object({…}))	✓
iam	Dataplex lake IAM bindings in {ROLE => [MEMBERS]} format.	map(list(string))		{}
location_type	The location type of the Dataplax Lake.	string		"SINGLE_REGION"
prefix	Optional prefix used to generate Dataplex Lake.	string		null

Outputs

name	description	sensitive
assets	Assets attached to the lake of Dataplex Lake.
id	Fully qualified Dataplex Lake id.
lake	The lake name of Dataplex Lake.
zones	The zone name of Dataplex Lake.