-
Notifications
You must be signed in to change notification settings - Fork 917
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add folder factory to project-factory module (#2152)
* WIP Folder Factory * parent keys and general fixes * changes * update README and example test, add support for hierarchy projects --------- Co-authored-by: Ludo <[email protected]>
- Loading branch information
Showing
7 changed files
with
233 additions
and
321 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,33 +1,57 @@ | ||
| # Project Factory | ||
| # Project and Folder Factory | ||
|
|
||
| This module implements in code the end-to-end project creation process for multiple projects via YAML data configurations. | ||
| This module implements end-to-end creation processes for a folder hierarchy, projects and billing budgets via YAML data configurations. | ||
|
|
||
| It supports | ||
|
|
||
| - filesystem-driven folder hierarchy exposing the full configuration options available in the [folder module](../folder/) | ||
| - multiple project creation and management exposing the full configuration options available in the [project module](../project/), including KMS key grants and VPC-SC perimeter membership | ||
| - optional per-project [service account management](#service-accounts) including basic IAM grants | ||
| - optional [billing budgets](#billing-budgets) factory and budget/project associations | ||
| - cross-referencing of hierarchy folders in projects | ||
| - optional per-project IaC configuration (TODO) | ||
|
|
||
| The factory is implemented as a thin wrapping layer, so that no "magic" or hidden side effects are implemented in code, and debugging or integration of new features are simple. | ||
| The factory is implemented as a thin data translation layer for the underlying modules, so that no "magic" or hidden side effects are implemented in code, and debugging or integration of new features are simple. | ||
|
|
||
| The code is meant to be executed by a high level service accounts with powerful permissions: | ||
|
|
||
| - Shared VPC connection if service project attachment is desired | ||
| - forlder admin permissions for the hierarchy | ||
| - project creation on the nodes (folder or org) where projects will be defined | ||
| - Shared VPC connection if service project attachment is desired | ||
| - billing cost manager permissions to manage budgets and monitoring permissions if notifications should also be managed here | ||
|
|
||
| <!-- BEGIN TOC --> | ||
| - [Leveraging data defaults, merges, optionals](#leveraging-data-defaults-merges-optionals) | ||
| - [Additional resources](#additional-resources) | ||
| - [Folder hierarchy](#folder-hierarchy) | ||
| - [Projects](#projects) | ||
| - [Leveraging project defaults, merges, optionals](#leveraging-project-defaults-merges-optionals) | ||
| - [Service accounts](#service-accounts) | ||
| - [Billing budgets](#billing-budgets) | ||
| - [Billing budgets](#billing-budgets) | ||
| - [Example](#example) | ||
| - [Variables](#variables) | ||
| - [Outputs](#outputs) | ||
| - [Tests](#tests) | ||
| <!-- END TOC --> | ||
|
|
||
| ## Leveraging data defaults, merges, optionals | ||
| ## Folder hierarchy | ||
|
|
||
| The hierarchy supports up to three levels of folders, which are defined via filesystem directories each including a `_config.yaml` files detailing their attributes. | ||
|
|
||
| The hierarchy factory is configured via the `factories_config.hierarchy` variable via one mandatory and one optional argument: | ||
|
|
||
| - `factories_config.hierarchy.folders_data_path` is required to enable the hierarchy factory, and must be set to the path containing the YAML definitions | ||
| - `factories_config.hierarchy.parent_ids` is an optional map where keys are arbitrary and values are set to resource node ids | ||
|
|
||
| Top-level folders in the filesystem hierarchy have no explicit parent, so their parent ids need to be provided in the YAML by either referencing the full id (e.g. `folders/12345678`) or by referencing a key in the `parent_ids` attribute described above. As a shortcut, a `default` key can be defined whose value is used for any top-level folder which does not directly provide a parent id. | ||
|
|
||
| Filesystem directories can also contain project definitions in the same YAML format described below. This approach must be used with caution and is best adopted for stable scenarios, as problems in the filesystem hierarchy definitions might result in the project files not being read and the resources being deleted by Terraform. | ||
|
|
||
| Refer to the [example](#example) below for actual examples of the YAML definitions. | ||
|
|
||
| ## Projects | ||
|
|
||
| The project factory is configured via the `factories_config.projects_data_path` variable, and project files are also read from the hierarchy describe in the previous section when enabled. The YAML format mirrors the project module, refer to the [example](#example) below for actual examples of the YAML definitions. | ||
|
|
||
| ### Leveraging project defaults, merges, optionals | ||
|
|
||
| In addition to the YAML-based project configurations, the factory accepts three additional sets of inputs via Terraform variables: | ||
|
|
||
|
|
@@ -37,8 +61,6 @@ In addition to the YAML-based project configurations, the factory accepts three | |
|
|
||
| Some examples on where to use each of the three sets are [provided below](#example). | ||
|
|
||
| ## Additional resources | ||
|
|
||
| ### Service accounts | ||
|
|
||
| Service accounts can be managed as part of each project's YAML configuration. This allows creation of default service accounts used for GCE instances, in firewall rules, or for application-level credentials without resorting to a separate Terraform configuration. | ||
|
|
@@ -59,9 +81,9 @@ service_accounts: | |
| Both the `display_name` and `iam_self_roles` attributes are optional. | ||
|
|
||
| ### Billing budgets | ||
| ## Billing budgets | ||
|
|
||
| The project factory integrates the billing budgets factory exposed by the `[`billing-account`](../billing-account/) module, and adds support for easy referencing budgets in project files. | ||
| The billing budgets factory integrates the `[`billing-account`](../billing-account/) module functionality, and adds support for easy referencing budgets in project files. | ||
|
|
||
| To enable support for billing budgets, set the billing account id, optional notification channels, and the data folder for budgets in the `factories_config.budgets` variable, then create billing budgets using YAML definitions following the format described in the `billing-account` module. | ||
|
|
||
|
|
@@ -84,6 +106,8 @@ The example below shows how to use the billing budgets factory. | |
|
|
||
| ## Example | ||
|
|
||
| The module invocation using all optional features: | ||
|
|
||
| ```hcl | ||
| module "project-factory" { | ||
| source = "./fabric/modules/project-factory" | ||
|
|
@@ -122,12 +146,56 @@ module "project-factory" { | |
| } | ||
| } | ||
| } | ||
| hierarchy = { | ||
| folders_data_path = "data/hierarchy" | ||
| parent_ids = { | ||
| default = "folders/12345678" | ||
| } | ||
| } | ||
| projects_data_path = "data/projects" | ||
| } | ||
| } | ||
| # tftest modules=8 resources=37 files=prj-app-1,prj-app-2,prj-app-3,budget-test-100 | ||
| # tftest modules=13 resources=48 files=prj-app-1,prj-app-2,prj-app-3,budget-test-100,h-0-0,h-1-0,h-0-1,h-1-1,h-1-1-p0 inventory=example.yaml | ||
| ``` | ||
|
|
||
| A simple hierarchy of folders: | ||
|
|
||
| ```yaml | ||
| name: Foo (level 1) | ||
| iam: | ||
| roles/viewer: | ||
| - group:[email protected] | ||
| # tftest-file id=h-0-0 path=data/hierarchy/foo/_config.yaml | ||
| ``` | ||
|
|
||
| ```yaml | ||
| name: Bar (level 1) | ||
| parent: folders/4567890 | ||
| # tftest-file id=h-1-0 path=data/hierarchy/bar/_config.yaml | ||
| ``` | ||
|
|
||
| ```yaml | ||
| name: Foo Baz (level 2) | ||
| # tftest-file id=h-0-1 path=data/hierarchy/foo/baz/_config.yaml | ||
| ``` | ||
|
|
||
| ```yaml | ||
| name: Bar Baz (level 2) | ||
| # tftest-file id=h-1-1 path=data/hierarchy/bar/baz/_config.yaml | ||
| ``` | ||
|
|
||
| One project defined within the folder hierarchy: | ||
|
|
||
| ```yaml | ||
| billing_account: 012345-67890A-BCDEF0 | ||
| services: | ||
| - container.googleapis.com | ||
| - storage.googleapis.com | ||
| # tftest-file id=h-1-1-p0 path=data/hierarchy/bar/baz/bar-baz-0.yaml | ||
| ``` | ||
|
|
||
| More traditional project definitions via the project factory data: | ||
|
|
||
| ```yaml | ||
| # project app-1 | ||
| billing_account: 012345-67890A-BCDEF0 | ||
|
|
@@ -206,6 +274,8 @@ services: | |
| # tftest-file id=prj-app-3 path=data/projects/prj-app-3.yaml | ||
| ``` | ||
|
|
||
| And a billing budget: | ||
|
|
||
| ```yaml | ||
| # billing budget test-100 | ||
| display_name: 100 dollars in current spend | ||
|
|
@@ -226,12 +296,13 @@ update_rules: | |
| - billing-default | ||
| # tftest-file id=budget-test-100 path=data/budgets/test-100.yaml | ||
| ``` | ||
|
|
||
| <!-- BEGIN TFDOC --> | ||
| ## Variables | ||
|
|
||
| | name | description | type | required | default | | ||
| |---|---|:---:|:---:|:---:| | ||
| | [factories_config](variables.tf#L91) | Path to folder with YAML resource description data files. | <code title="object({ projects_data_path = string budgets = optional(object({ billing_account = string budgets_data_path = string notification_channels = optional(map(any), {}) })) })">object({…})</code> | ✓ | | | ||
| | [factories_config](variables.tf#L91) | Path to folder with YAML resource description data files. | <code title="object({ hierarchy = optional(object({ folders_data_path = string parent_ids = optional(map(string), {}) })) projects_data_path = optional(string) budgets = optional(object({ billing_account = string budgets_data_path = string notification_channels = optional(map(any), {}) })) })">object({…})</code> | ✓ | | | ||
| | [data_defaults](variables.tf#L17) | Optional default values used when corresponding project data from files are missing. | <code title="object({ billing_account = optional(string) contacts = optional(map(list(string)), {}) labels = optional(map(string), {}) metric_scopes = optional(list(string), []) parent = optional(string) prefix = optional(string) service_encryption_key_ids = optional(map(list(string)), {}) service_perimeter_bridges = optional(list(string), []) service_perimeter_standard = optional(string) services = optional(list(string), []) shared_vpc_service_config = optional(object({ host_project = string network_users = optional(list(string), []) service_identity_iam = optional(map(list(string)), {}) service_identity_subnet_iam = optional(map(list(string)), {}) service_iam_grants = optional(list(string), []) network_subnet_users = optional(map(list(string)), {}) }), { host_project = null }) tag_bindings = optional(map(string), {}) service_accounts = optional(map(object({ display_name = optional(string, "Terraform-managed.") iam_self_roles = optional(list(string)) })), {}) })">object({…})</code> | | <code>{}</code> | | ||
| | [data_merges](variables.tf#L49) | Optional values that will be merged with corresponding data from files. Combines with `data_defaults`, file data, and `data_overrides`. | <code title="object({ contacts = optional(map(list(string)), {}) labels = optional(map(string), {}) metric_scopes = optional(list(string), []) service_encryption_key_ids = optional(map(list(string)), {}) service_perimeter_bridges = optional(list(string), []) services = optional(list(string), []) tag_bindings = optional(map(string), {}) service_accounts = optional(map(object({ display_name = optional(string, "Terraform-managed.") iam_self_roles = optional(list(string)) })), {}) })">object({…})</code> | | <code>{}</code> | | ||
| | [data_overrides](variables.tf#L69) | Optional values that override corresponding data from files. Takes precedence over file data and `data_defaults`. | <code title="object({ billing_account = optional(string) contacts = optional(map(list(string))) parent = optional(string) prefix = optional(string) service_encryption_key_ids = optional(map(list(string))) service_perimeter_bridges = optional(list(string)) service_perimeter_standard = optional(string) tag_bindings = optional(map(string)) services = optional(list(string)) service_accounts = optional(map(object({ display_name = optional(string, "Terraform-managed.") iam_self_roles = optional(list(string)) }))) })">object({…})</code> | | <code>{}</code> | | ||
|
|
@@ -240,8 +311,9 @@ update_rules: | |
|
|
||
| | name | description | sensitive | | ||
| |---|---|:---:| | ||
| | [projects](outputs.tf#L17) | Project module outputs. | | | ||
| | [service_accounts](outputs.tf#L22) | Service account emails. | | | ||
| | [folders](outputs.tf#L17) | Folder ids. | | | ||
| | [projects](outputs.tf#L22) | Project module outputs. | | | ||
| | [service_accounts](outputs.tf#L27) | Service account emails. | | | ||
| <!-- END TFDOC --> | ||
| ## Tests | ||
|
|
||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,103 @@ | ||
| /** | ||
| * Copyright 2024 Google LLC | ||
| * | ||
| * Licensed under the Apache License, Version 2.0 (the "License"); | ||
| * you may not use this file except in compliance with the License. | ||
| * You may obtain a copy of the License at | ||
| * | ||
| * http://www.apache.org/licenses/LICENSE-2.0 | ||
| * | ||
| * Unless required by applicable law or agreed to in writing, software | ||
| * distributed under the License is distributed on an "AS IS" BASIS, | ||
| * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| * See the License for the specific language governing permissions and | ||
| * limitations under the License. | ||
| */ | ||
|
|
||
| locals { | ||
| _folders_path = try( | ||
| pathexpand(var.factories_config.hierarchy.folders_data_path), null | ||
| ) | ||
| _folders = { | ||
| for f in local._hierarchy_files : dirname(f) => yamldecode(file( | ||
| "${coalesce(var.factories_config.hierarchy.folders_data_path, "-")}/${f}" | ||
| )) | ||
| } | ||
| _hierarchy_files = try( | ||
| fileset(local._folders_path, "**/_config.yaml"), | ||
| [] | ||
| ) | ||
| folders = { | ||
| for key, data in local._folders : key => merge(data, { | ||
| key = key | ||
| level = length(split("/", key)) | ||
| parent_key = dirname(key) | ||
| }) | ||
| } | ||
| hierarchy = merge( | ||
| try(var.factories_config.hierarchy.parent_ids, {}), | ||
| { for k, v in module.hierarchy-folder-lvl-1 : k => v.id }, | ||
| { for k, v in module.hierarchy-folder-lvl-2 : k => v.id }, | ||
| { for k, v in module.hierarchy-folder-lvl-3 : k => v.id }, | ||
| ) | ||
| } | ||
|
|
||
| check "hierarchy-data" { | ||
| assert { | ||
| condition = ( | ||
| var.factories_config.hierarchy == null || | ||
| try(var.factories_config.hierarchy.parent_ids.default, null) != null | ||
| ) | ||
| error_message = "No default set for hierarchy parent ids." | ||
| } | ||
| } | ||
|
|
||
| module "hierarchy-folder-lvl-1" { | ||
| source = "../folder" | ||
| for_each = { for k, v in local.folders : k => v if v.level == 1 } | ||
| parent = try( | ||
| # allow the YAML data to set the parent for this level | ||
| lookup( | ||
| var.factories_config.hierarchy.parent_ids, | ||
| each.value.parent, | ||
| # use the value as is if it's not in the parents map | ||
| each.value.parent | ||
| ), | ||
| # use the default value in the initial parents map | ||
| var.factories_config.hierarchy.parent_ids.default | ||
| # fail if we don't have an explicit parent | ||
| ) | ||
| name = each.value.name | ||
| iam = lookup(each.value, "iam", {}) | ||
| iam_bindings = lookup(each.value, "iam_bindings", {}) | ||
| iam_bindings_additive = lookup(each.value, "iam_bindings_additive", {}) | ||
| iam_by_principals = lookup(each.value, "iam_by_principals", {}) | ||
| org_policies = lookup(each.value, "org_policies", {}) | ||
| tag_bindings = lookup(each.value, "tag_bindings", {}) | ||
| } | ||
|
|
||
| module "hierarchy-folder-lvl-2" { | ||
| source = "../folder" | ||
| for_each = { for k, v in local.folders : k => v if v.level == 2 } | ||
| parent = module.hierarchy-folder-lvl-1[each.value.parent_key].id | ||
| name = each.value.name | ||
| iam = lookup(each.value, "iam", {}) | ||
| iam_bindings = lookup(each.value, "iam_bindings", {}) | ||
| iam_bindings_additive = lookup(each.value, "iam_bindings_additive", {}) | ||
| iam_by_principals = lookup(each.value, "iam_by_principals", {}) | ||
| org_policies = lookup(each.value, "org_policies", {}) | ||
| tag_bindings = lookup(each.value, "tag_bindings", {}) | ||
| } | ||
|
|
||
| module "hierarchy-folder-lvl-3" { | ||
| source = "../folder" | ||
| for_each = { for k, v in local.folders : k => v if v.level == 3 } | ||
| parent = module.hierarchy-folder-lvl-2[each.value.parent_key].id | ||
| name = each.value.name | ||
| iam = lookup(each.value, "iam", {}) | ||
| iam_bindings = lookup(each.value, "iam_bindings", {}) | ||
| iam_bindings_additive = lookup(each.value, "iam_bindings_additive", {}) | ||
| iam_by_principals = lookup(each.value, "iam_by_principals", {}) | ||
| org_policies = lookup(each.value, "org_policies", {}) | ||
| tag_bindings = lookup(each.value, "tag_bindings", {}) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.