Support automation/controlling projects and resources in project factory

modules/project-factory/README.md

@@ -23,10 +23,12 @@ The code is meant to be executed by a high level service accounts with powerful
 <!-- BEGIN TOC -->
 - [Folder hierarchy](#folder-hierarchy)
 - [Projects](#projects)
-  - [Leveraging project defaults, merges, optionals](#leveraging-project-defaults-merges-optionals)
+  - [Factory-wide project defaults, merges, optionals](#factory-wide-project-defaults-merges-optionals)
   - [Service accounts](#service-accounts)
+  - [Automation project and resources](#automation-project-and-resources)
 - [Billing budgets](#billing-budgets)
 - [Example](#example)
+- [Files](#files)
 - [Variables](#variables)
 - [Outputs](#outputs)
 - [Tests](#tests)
@@ -51,7 +53,7 @@ Refer to the [example](#example) below for actual examples of the YAML definitio
 
 The project factory is configured via the `factories_config.projects_data_path` variable, and project files are also read from the hierarchy describe in the previous section when enabled. The YAML format mirrors the project module, refer to the [example](#example) below for actual examples of the YAML definitions.
 
-### Leveraging project defaults, merges, optionals
+### Factory-wide project defaults, merges, optionals
 
 In addition to the YAML-based project configurations, the factory accepts three additional sets of inputs via Terraform variables:
 
@@ -81,6 +83,54 @@ service_accounts:
 
 Both the `display_name` and `iam_self_roles` attributes are optional.
 
+### Automation project and resources
+
+Project configurations also support defining service accounts and storage buckets to support automation, created in a separate controlling project so as to be outside of the sphere of control of the managed project.
+
+Automation resources are defined via the `automation` attribute in project configurations, which supports:
+
+- a mandatory `project` attribute to define the external controlling project
+- an optional `service_accounts` list where each element will define a service account in the controlling project
+- an optional `buckets` map where each key will define a bucket in the controlling project, and the map of roles/principals in the corresponding value assigned on the created bucket; principals can refer to the created service accounts by key
+
+Service accounts and buckets will be prefixed with the project name, and use the key specified in the YAML file as a suffix.
+
+```yaml
+# file name: prod-app-example-0
+# prefix via factory defaults: foo
+# project id: foo-prod-app-example-0
+billing_account: 012345-67890A-BCDEF0
+parent: folders/12345678
+services:
+  - compute.googleapis.com
+  - stackdriver.googleapis.com
+iam:
+  roles/owner:
+    - rw
+  roles/viewer:
+    - ro
+automation:
+  project: foo-prod-iac-core-0
+  service_accounts:
+    # sa name: foo-prod-app-example-0-rw
+    rw:
+      description: Read/write automation sa for app example 0.
+    # sa name: foo-prod-app-example-0-ro
+    ro:
+      description: Read-only automation sa for app example 0.
+  buckets:
+    # bucket name: foo-prod-app-example-0-state
+    state:
+      description: Terraform state bucket for app example 0.
+      iam:
+        roles/storage.objectCreator:
+          - rw
+        roles/storage.objectViewer:
+          - rw
+          - ro
+          - "group: [email protected]"
+```
+
 ## Billing budgets
 
 The billing budgets factory integrates the `[`billing-account`](../billing-account/) module functionality, and adds support for easy referencing budgets in project files.
@@ -102,7 +152,7 @@ billing_budgets:
   - test-100
 ```
 
-The example below shows how to use the billing budgets factory.
+A simple billing budget example is show in the [example](#example) below.
 
 ## Example
 
@@ -155,7 +205,7 @@ module "project-factory" {
     projects_data_path = "data/projects"
   }
 }
-# tftest modules=13 resources=48 files=prj-app-1,prj-app-2,prj-app-3,budget-test-100,h-0-0,h-1-0,h-0-1,h-1-1,h-1-1-p0 inventory=example.yaml
+# tftest modules=16 resources=55 files=prj-app-1,prj-app-2,prj-app-3,budget-test-100,h-0-0,h-1-0,h-0-1,h-1-1,h-1-1-p0 inventory=example.yaml
 ```
 
 A simple hierarchy of folders:
@@ -191,7 +241,7 @@ billing_account: 012345-67890A-BCDEF0
 services:
   - container.googleapis.com
   - storage.googleapis.com
-# tftest-file id=h-1-1-p0 path=data/hierarchy/bar/baz/bar-baz-0.yaml
+# tftest-file id=h-1-1-p0 path=data/hierarchy/bar/baz/bar-baz-iac-0.yaml
 ```
 
 More traditional project definitions via the project factory data:
@@ -264,12 +314,36 @@ shared_vpc_service_config:
 # tftest-file id=prj-app-2 path=data/projects/prj-app-2.yaml
 ```
 
+This project uses a reference to a hierarchy folder, and defines a controlling project via the `automation` attributes:
+
 ```yaml
-# project app-3
-parent: folders/12345678
+parent: bar/baz
 services:
 - run.googleapis.com
 - storage.googleapis.com
+iam:
+  "roles/owner":
+    - rw
+  "roles/viewer":
+    - ro
+automation:
+  project: bar-baz-iac-0
+  service_accounts:
+    rw:
+      description: Read/write automation sa for app example 0.
+    ro:
+      description: Read-only automation sa for app example 0.
+  buckets:
+    state:
+      description: Terraform state bucket for app example 0.
+      iam:
+        roles/storage.objectCreator:
+          - rw
+        roles/storage.objectViewer:
+          - rw
+          - ro
+          - "group: [email protected]"
+
 
 # tftest-file id=prj-app-3 path=data/projects/prj-app-3.yaml
 ```
@@ -297,7 +371,21 @@ update_rules:
 # tftest-file id=budget-test-100 path=data/budgets/test-100.yaml
 ```
 
+<!-- TFDOC OPTS files:1 -->
 <!-- BEGIN TFDOC -->
+## Files
+
+| name | description | modules |
+|---|---|---|
+| [automation.tf](./automation.tf) | Automation projects locals and resources. | <code>gcs</code> · <code>iam-service-account</code> |
+| [factory-budgets.tf](./factory-budgets.tf) | Billing budget factory locals. |  |
+| [factory-folders.tf](./factory-folders.tf) | Folder hierarchy factory locals. |  |
+| [factory-projects.tf](./factory-projects.tf) | Projects factory locals. |  |
+| [folders.tf](./folders.tf) | Folder hierarchy factory resources. | <code>folder</code> |
+| [main.tf](./main.tf) | Projects and billing budgets factory resources. | <code>billing-account</code> · <code>iam-service-account</code> · <code>project</code> |
+| [outputs.tf](./outputs.tf) | Module outputs. |  |
+| [variables.tf](./variables.tf) | Module variables. |  |
+
 ## Variables
 
 | name | description | type | required | default |

modules/project-factory/automation.tf

@@ -0,0 +1,109 @@
+/**
+ * Copyright 2024 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+# tfdoc:file:description Automation projects locals and resources.
+
+locals {
+  automation_buckets = flatten([
+    for k, v in local.projects : [
+      for ks, kv in try(v.automation.buckets, {}) : merge(kv, {
+        automation_project = v.automation.project
+        name               = ks
+        prefix             = v.prefix
+        project            = k
+      })
+    ]
+  ])
+  automation_sa = flatten([
+    for k, v in local.projects : [
+      for ks, kv in try(v.automation.service_accounts, {}) : merge(kv, {
+        automation_project = v.automation.project
+        name               = ks
+        prefix             = v.prefix
+        project            = k
+      })
+    ]
+  ])
+}
+
+module "automation-buckets" {
+  source = "../gcs"
+  for_each = {
+    for k in local.automation_buckets : "${k.project}/${k.name}" => k
+  }
+  project_id     = each.value.automation_project
+  prefix         = each.value.prefix
+  name           = "${each.value.project}-${each.value.name}"
+  encryption_key = lookup(each.value, "encryption_key", null)
+  # try interpolating service accounts by key in principals
+  iam = {
+    for k, v in lookup(each.value, "iam", {}) : k => [
+      for vv in v : try(
+        module.automation-service-accounts["${each.value.project}/${vv}"].iam_email,
+        vv
+      )
+    ]
+  }
+  iam_bindings = {
+    for k, v in lookup(each.value, "iam_bindings", {}) : k => merge(v, {
+      members = [
+        for vv in v.members : try(
+          module.automation-service-accounts["${each.value.project}/${vv}"].iam_email,
+          vv
+        )
+      ]
+    })
+  }
+  iam_bindings_additive = {
+    for k, v in lookup(each.value, "iam_bindings_additive", {}) : k => merge(v, {
+      member = try(
+        module.automation-service-accounts["${each.value.project}/${v.member}"].iam_email,
+        v.member
+      )
+    })
+  }
+  labels                      = lookup(each.value, "labels", {})
+  location                    = lookup(each.value, "location", "EU")
+  storage_class               = lookup(each.value, "storage_class", "MULTI_REGIONAL")
+  uniform_bucket_level_access = lookup(each.value, "uniform_bucket_level_access", true)
+  versioning                  = lookup(each.value, "versioning", false)
+}
+
+module "automation-service-accounts" {
+  source = "../iam-service-account"
+  for_each = {
+    for k in local.automation_sa : "${k.project}/${k.name}" => k
+  }
+  project_id  = each.value.automation_project
+  prefix      = each.value.prefix
+  name        = "${each.value.project}-${each.value.name}"
+  description = lookup(each.value, "description", null)
+  display_name = lookup(
+    each.value,
+    "display_name",
+    "Service account ${each.value.name} for ${each.value.project}."
+  )
+  iam                    = lookup(each.value, "iam", {})
+  iam_bindings           = lookup(each.value, "iam_bindings", {})
+  iam_bindings_additive  = lookup(each.value, "iam_bindings_additive", {})
+  iam_billing_roles      = lookup(each.value, "iam_billing_roles", {})
+  iam_folder_roles       = lookup(each.value, "iam_folder_roles", {})
+  iam_organization_roles = lookup(each.value, "iam_organization_roles", {})
+  iam_project_roles      = lookup(each.value, "iam_project_roles", {})
+  iam_sa_roles           = lookup(each.value, "iam_sa_roles", {})
+  # we don't interpolate buckets here as we can't use a dynamic key
+  iam_storage_roles = lookup(each.value, "iam_storage_roles", {})
+}

modules/project-factory/factory-budgets.tf

@@ -14,6 +14,8 @@
  * limitations under the License.
  */
 
+# tfdoc:file:description Billing budget factory locals.
+
 locals {
   # reimplement the billing account factory here to interpolate projects
   _budget_path = try(pathexpand(var.factories_config.budgets.budgets_data_path), null)

modules/project-factory/factory-folders.tf

@@ -14,6 +14,8 @@
  * limitations under the License.
  */
 
+# tfdoc:file:description Folder hierarchy factory locals.
+
 locals {
   _folders_path = try(
     pathexpand(var.factories_config.hierarchy.folders_data_path), null
@@ -51,53 +53,3 @@ check "hierarchy-data" {
     error_message = "No default set for hierarchy parent ids."
   }
 }
-
-module "hierarchy-folder-lvl-1" {
-  source   = "../folder"
-  for_each = { for k, v in local.folders : k => v if v.level == 1 }
-  parent = try(
-    # allow the YAML data to set the parent for this level
-    lookup(
-      var.factories_config.hierarchy.parent_ids,
-      each.value.parent,
-      # use the value as is if it's not in the parents map
-      each.value.parent
-    ),
-    # use the default value in the initial parents map
-    var.factories_config.hierarchy.parent_ids.default
-    # fail if we don't have an explicit parent
-  )
-  name                  = each.value.name
-  iam                   = lookup(each.value, "iam", {})
-  iam_bindings          = lookup(each.value, "iam_bindings", {})
-  iam_bindings_additive = lookup(each.value, "iam_bindings_additive", {})
-  iam_by_principals     = lookup(each.value, "iam_by_principals", {})
-  org_policies          = lookup(each.value, "org_policies", {})
-  tag_bindings          = lookup(each.value, "tag_bindings", {})
-}
-
-module "hierarchy-folder-lvl-2" {
-  source                = "../folder"
-  for_each              = { for k, v in local.folders : k => v if v.level == 2 }
-  parent                = module.hierarchy-folder-lvl-1[each.value.parent_key].id
-  name                  = each.value.name
-  iam                   = lookup(each.value, "iam", {})
-  iam_bindings          = lookup(each.value, "iam_bindings", {})
-  iam_bindings_additive = lookup(each.value, "iam_bindings_additive", {})
-  iam_by_principals     = lookup(each.value, "iam_by_principals", {})
-  org_policies          = lookup(each.value, "org_policies", {})
-  tag_bindings          = lookup(each.value, "tag_bindings", {})
-}
-
-module "hierarchy-folder-lvl-3" {
-  source                = "../folder"
-  for_each              = { for k, v in local.folders : k => v if v.level == 3 }
-  parent                = module.hierarchy-folder-lvl-2[each.value.parent_key].id
-  name                  = each.value.name
-  iam                   = lookup(each.value, "iam", {})
-  iam_bindings          = lookup(each.value, "iam_bindings", {})
-  iam_bindings_additive = lookup(each.value, "iam_bindings_additive", {})
-  iam_by_principals     = lookup(each.value, "iam_by_principals", {})
-  org_policies          = lookup(each.value, "org_policies", {})
-  tag_bindings          = lookup(each.value, "tag_bindings", {})
-}

modules/project-factory/factory-projects.tf

@@ -13,6 +13,9 @@
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */
+
+# tfdoc:file:description Projects factory locals.
+
 locals {
   _hierarchy_projects = (
     {