Merge pull request #801 from GoogleCloudPlatform/lcaggio/cloudsql-shared

Update Cloud SQL example

examples/data-solutions/README.md

@@ -30,7 +30,7 @@ This [example](./data-platform-foundations/) implements SQL Server Always On Ava
 
 ### Cloud SQL instance with multi-region read replicas
 
-<a href="./cloudsql-multiregion/" title="Cloud SQL instance with multi-region read replicas"><img src="./cloudsql-multiregion/diagram.png" align="left" width="280px"></a>
+<a href="./cloudsql-multiregion/" title="Cloud SQL instance with multi-region read replicas"><img src="./cloudsql-multiregion/images/diagram.png" align="left" width="280px"></a>
 This [example](./cloudsql-multiregion/) creates a [Cloud SQL instance](https://cloud.google.com/sql) with multi-region read replicas as described in the [Cloud SQL for PostgreSQL disaster recovery](https://cloud.google.com/architecture/cloud-sql-postgres-disaster-recovery-complete-failover-fallback) article.
 <br clear="left">
 

examples/data-solutions/cloudsql-multiregion/README.md

@@ -80,6 +80,27 @@ This implementation is intentionally minimal and easy to read. A real world use
 - Using a Shared VPC
 - Using VPC-SC to mitigate data exfiltration
 
+### Shared VPC
+The example supports the configuration of a Shared VPC as an input variable. 
+To deploy the solution on a Shared VPC, you have to configure the `network_config` variable:
+
+```
+network_config = {
+    host_project       = "PROJECT_ID"
+    network_self_link  = "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/global/networks/VPC_NAME"
+    subnet_self_link   = "https://www.googleapis.com/compute/v1/projects/PROJECT_ID/regions/$REGION/subnetworks/SUBNET_NAME"
+    cloudsql_psa_range = "10.60.0.0/24"
+  }
+```
+
+To run this example, the Shared VPC project needs to have:
+ - A Private Service Connect with a range of `/24` (example: `10.60.0.0/24`) to deploy the Cloud SQL instance.
+ - Internet access configured (for example Cloud NAT) to let the Test VM download packages. 
+
+In order to run the example and deploy Cloud SQL on a shared VPC the identity running Terraform must have the following IAM role on the Shared VPC Host project.
+ - Compute Network Admin (roles/compute.networkAdmin)
+ - Compute Shared VPC Admin (roles/compute.xpnAdmin)
+
 ## Test your environment
 
 We assume all those steps are run using a user listed on `data_eng_principals`. You can authenticate as the user using the following command:
@@ -118,15 +139,16 @@ The above command will delete the associated resources so there will be no billa
 
 | name | description | type | required | default |
 |---|---|:---:|:---:|:---:|
-| [postgres_user_password](variables.tf#L29) | `postgres` user password. | <code>string</code> | ✓ |  |
-| [prefix](variables.tf#L40) | Unique prefix used for resource names. Not used for project if 'project_create' is null. | <code>string</code> | ✓ |  |
-| [project_id](variables.tf#L54) | Project id, references existing project if `project_create` is null. | <code>string</code> | ✓ |  |
-| [cmek_encryption](variables.tf#L17) | Flag to enable CMEK on GCP resources created. | <code>bool</code> |  | <code>false</code> |
+| [postgres_user_password](variables.tf#L40) | `postgres` user password. | <code>string</code> | ✓ |  |
+| [prefix](variables.tf#L51) | Unique prefix used for resource names. Not used for project if 'project_create' is null. | <code>string</code> | ✓ |  |
+| [project_id](variables.tf#L65) | Project id, references existing project if `project_create` is null. | <code>string</code> | ✓ |  |
 | [data_eng_principals](variables.tf#L23) | Groups with Service Account Token creator role on service accounts in IAM format, only user supported on CloudSQL, eg '[email protected]'. | <code>list&#40;string&#41;</code> |  | <code>&#91;&#93;</code> |
-| [postgres_database](variables.tf#L34) | `postgres` database. | <code>string</code> |  | <code>&#34;guestbook&#34;</code> |
-| [project_create](variables.tf#L45) | Provide values if project creation is needed, uses existing project if null. Parent is in 'folders/nnn' or 'organizations/nnn' format. | <code title="object&#40;&#123;&#10;  billing_account_id &#61; string&#10;  parent             &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code>null</code> |
-| [regions](variables.tf#L59) | Map of instance_name => location where instances will be deployed. | <code>map&#40;string&#41;</code> |  | <code title="&#123;&#10;  primary &#61; &#34;europe-west1&#34;&#10;  replica &#61; &#34;europe-west3&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
-| [sql_configuration](variables.tf#L73) | Cloud SQL configuration | <code title="object&#40;&#123;&#10;  availability_type &#61; string&#10;  database_version  &#61; string&#10;  psa_range         &#61; string&#10;  tier              &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code title="&#123;&#10;  availability_type &#61; &#34;REGIONAL&#34;&#10;  database_version  &#61; &#34;POSTGRES_13&#34;&#10;  psa_range         &#61; &#34;10.60.0.0&#47;16&#34;&#10;  tier              &#61; &#34;db-g1-small&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
+| [network_config](variables.tf#L29) | Shared VPC network configurations to use. If null networks will be created in projects with preconfigured values. | <code title="object&#40;&#123;&#10;  host_project       &#61; string&#10;  network_self_link  &#61; string&#10;  subnet_self_link   &#61; string&#10;  cloudsql_psa_range &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code>null</code> |
+| [postgres_database](variables.tf#L45) | `postgres` database. | <code>string</code> |  | <code>&#34;guestbook&#34;</code> |
+| [project_create](variables.tf#L56) | Provide values if project creation is needed, uses existing project if null. Parent is in 'folders/nnn' or 'organizations/nnn' format. | <code title="object&#40;&#123;&#10;  billing_account_id &#61; string&#10;  parent             &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code>null</code> |
+| [regions](variables.tf#L70) | Map of instance_name => location where instances will be deployed. | <code>map&#40;string&#41;</code> |  | <code title="&#123;&#10;  primary &#61; &#34;europe-west1&#34;&#10;  replica &#61; &#34;europe-west3&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
+| [service_encryption_keys](variables.tf#L17) | Cloud KMS keys to use to encrypt resources. Provide a key for each reagion configured. | <code>map&#40;string&#41;</code> |  | <code>null</code> |
+| [sql_configuration](variables.tf#L84) | Cloud SQL configuration | <code title="object&#40;&#123;&#10;  availability_type &#61; string&#10;  database_version  &#61; string&#10;  psa_range         &#61; string&#10;  tier              &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code title="&#123;&#10;  availability_type &#61; &#34;REGIONAL&#34;&#10;  database_version  &#61; &#34;POSTGRES_13&#34;&#10;  psa_range         &#61; &#34;10.60.0.0&#47;16&#34;&#10;  tier              &#61; &#34;db-g1-small&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
 
 ## Outputs
 

examples/data-solutions/cloudsql-multiregion/cloudsql.tf

@@ -16,8 +16,8 @@ module "db" {
   source              = "../../../modules/cloudsql-instance"
   project_id          = module.project.project_id
   availability_type   = var.sql_configuration.availability_type
-  encryption_key_name = var.cmek_encryption ? module.kms[var.regions.primary].keys.key.id : null
-  network             = module.vpc.self_link
+  encryption_key_name = var.service_encryption_keys != null ? try(var.service_encryption_keys[var.regions.primary], null) : null
+  network             = local.vpc_self_link
   name                = "${var.prefix}-db"
   region              = var.regions.primary
   database_version    = var.sql_configuration.database_version
@@ -29,7 +29,7 @@ module "db" {
     for k, v in var.regions :
     k => {
       region              = v,
-      encryption_key_name = var.cmek_encryption ? module.kms[v].keys.key.id : null
+      encryption_key_name = var.service_encryption_keys != null ? try(var.service_encryption_keys[v], null) : null
     } if k != "primary"
   }
   databases = [var.postgres_database]

examples/data-solutions/cloudsql-multiregion/datastorage.tf

@@ -19,7 +19,7 @@ module "gcs" {
   name           = "data"
   location       = var.regions.primary
   storage_class  = "REGIONAL"
-  encryption_key = var.cmek_encryption ? module.kms[var.regions.primary].keys["key"].id : null
+  encryption_key = var.service_encryption_keys != null ? try(var.service_encryption_keys[var.regions.primary], null) : null
   force_destroy  = true
 }
 

examples/data-solutions/cloudsql-multiregion/gce.tf

@@ -29,8 +29,8 @@ module "test-vm" {
   zone       = "${var.regions.primary}-b"
   name       = "sql-test"
   network_interfaces = [{
-    network    = module.vpc.self_link
-    subnetwork = module.vpc.subnets["${var.regions.primary}/subnet"].self_link
+    network    = local.vpc_self_link
+    subnetwork = local.subnet
     nat        = false
     addresses  = null
   }]
@@ -50,10 +50,10 @@ module "test-vm" {
     type  = "pd-ssd"
     size  = 10
   }
-  encryption = var.cmek_encryption ? {
+  encryption = var.service_encryption_keys != null ? {
     encrypt_boot            = true
     disk_encryption_key_raw = null
-    kms_key_self_link       = var.cmek_encryption ? module.kms[var.regions.primary].keys["key"].id : null
+    kms_key_self_link       = var.service_encryption_keys != null ? try(var.service_encryption_keys[var.regions.primary], null) : null
   } : null
   metadata = { startup-script = local.startup-script }
   tags     = ["ssh"]

examples/data-solutions/cloudsql-multiregion/main.tf

@@ -54,6 +54,20 @@ locals {
       "serviceAccount:${module.project.service_accounts.robots.sql}"
     ]
   }
+
+  shared_vpc_project = try(var.network_config.host_project, null)
+  use_shared_vpc     = var.network_config != null
+
+  subnet = (
+    local.use_shared_vpc
+    ? var.network_config.subnet_self_link
+    : values(module.vpc.0.subnet_self_links)[0]
+  )
+  vpc_self_link = (
+    local.use_shared_vpc
+    ? var.network_config.network_self_link
+    : module.vpc.0.self_link
+  )
 }
 
 module "project" {
@@ -67,6 +81,7 @@ module "project" {
   iam_additive    = var.project_create == null ? local.iam : {}
   services = [
     "cloudkms.googleapis.com",
+    "compute.googleapis.com",
     "iap.googleapis.com",
     "logging.googleapis.com",
     "monitoring.googleapis.com",
@@ -77,10 +92,26 @@ module "project" {
     "storage.googleapis.com",
     "storage-component.googleapis.com",
   ]
+
+  shared_vpc_service_config = local.shared_vpc_project == null ? null : {
+    attach               = true
+    host_project         = local.shared_vpc_project
+    service_identity_iam = {}
+  }
+
+  service_encryption_key_ids = {
+    compute = try(values(var.service_encryption_keys), [])
+    sql     = try(values(var.service_encryption_keys), [])
+    storage = try(values(var.service_encryption_keys), [])
+  }
+  service_config = {
+    disable_on_destroy = false, disable_dependent_services = false
+  }
 }
 
 module "vpc" {
   source     = "../../../modules/net-vpc"
+  count      = local.use_shared_vpc ? 0 : 1
   project_id = module.project.project_id
   name       = "vpc"
   subnets = [
@@ -100,15 +131,17 @@ module "vpc" {
 
 module "firewall" {
   source       = "../../../modules/net-vpc-firewall"
+  count        = local.use_shared_vpc ? 0 : 1
   project_id   = module.project.project_id
-  network      = module.vpc.name
+  network      = module.vpc.0.name
   admin_ranges = ["10.0.0.0/20"]
 }
 
 module "nat" {
   source         = "../../../modules/net-cloudnat"
+  count          = local.use_shared_vpc ? 0 : 1
   project_id     = module.project.project_id
   region         = var.regions.primary
   name           = "${var.prefix}-default"
-  router_network = module.vpc.name
+  router_network = module.vpc.0.name
 }

examples/data-solutions/cloudsql-multiregion/variables.tf

@@ -14,10 +14,10 @@
  * limitations under the License.
  */
 
-variable "cmek_encryption" {
-  description = "Flag to enable CMEK on GCP resources created."
-  type        = bool
-  default     = false
+variable "service_encryption_keys" {
+  description = "Cloud KMS keys to use to encrypt resources. Provide a key for each reagion configured."
+  type        = map(string)
+  default     = null
 }
 
 variable "data_eng_principals" {
@@ -26,6 +26,17 @@ variable "data_eng_principals" {
   default     = []
 }
 
+variable "network_config" {
+  description = "Shared VPC network configurations to use. If null networks will be created in projects with preconfigured values."
+  type = object({
+    host_project       = string
+    network_self_link  = string
+    subnet_self_link   = string
+    cloudsql_psa_range = string
+  })
+  default = null
+}
+
 variable "postgres_user_password" {
   description = "`postgres` user password."
   type        = string

tests/examples/data_solutions/cloudsql-multiregion/test_plan.py

@@ -15,5 +15,5 @@
 def test_resources(e2e_plan_runner):
   "Test that plan works and the numbers of resources is as expected."
   modules, resources = e2e_plan_runner()
-  assert len(modules) == 11
-  assert len(resources) == 53
+  assert len(modules) == 9
+  assert len(resources) == 48