-
Notifications
You must be signed in to change notification settings - Fork 916
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Add support for billing budgets to project factory (#2112)
* align factory variable name in project factory module * tested * align fast stage
- Loading branch information
Showing
14 changed files
with
253 additions
and
49 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| data |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,12 +4,10 @@ This module implements in code the end-to-end project creation process for multi | |
|
|
||
| It supports | ||
|
|
||
| - all project-level attributes exposed by the [project module](../project/), including Shared VPC host/service configuration | ||
| - optional service account creation in the project, including basic IAM grants | ||
| - KMS key encrypt/decrypt permissions for service identities in the project | ||
| - membership in VPC SC standard or bridge perimeters | ||
| - billing budgets (TODO) | ||
| - per-project IaC configuration (TODO) | ||
| - multiple project creation and management exposing the full configuration options available in the [project module](../project/), including KMS key grants and VPC-SC perimeter membership | ||
| - optional per-project [service account management](#service-accounts) including basic IAM grants | ||
| - optional [billing budgets](#billing-budgets) factory and budget/project associations | ||
| - optional per-project IaC configuration (TODO) | ||
|
|
||
| The factory is implemented as a thin wrapping layer, so that no "magic" or hidden side effects are implemented in code, and debugging or integration of new features are simple. | ||
|
|
||
|
|
@@ -18,6 +16,17 @@ The code is meant to be executed by a high level service accounts with powerful | |
| - Shared VPC connection if service project attachment is desired | ||
| - project creation on the nodes (folder or org) where projects will be defined | ||
|
|
||
| <!-- BEGIN TOC --> | ||
| - [Leveraging data defaults, merges, optionals](#leveraging-data-defaults-merges-optionals) | ||
| - [Additional resources](#additional-resources) | ||
| - [Service accounts](#service-accounts) | ||
| - [Billing budgets](#billing-budgets) | ||
| - [Example](#example) | ||
| - [Variables](#variables) | ||
| - [Outputs](#outputs) | ||
| - [Tests](#tests) | ||
| <!-- END TOC --> | ||
|
|
||
| ## Leveraging data defaults, merges, optionals | ||
|
|
||
| In addition to the YAML-based project configurations, the factory accepts three additional sets of inputs via Terraform variables: | ||
|
|
@@ -26,7 +35,49 @@ In addition to the YAML-based project configurations, the factory accepts three | |
| - the `data_overrides` variable works similarly to defaults, but the values specified here take precedence over those in YAML files | ||
| - the `data_merges` variable allows specifying additional values for map or set based variables, which are merged with the data coming from YAML | ||
|
|
||
| Some examples on where to use each of the three sets are provided below. | ||
| Some examples on where to use each of the three sets are [provided below](#example). | ||
|
|
||
| ## Additional resources | ||
|
|
||
| ### Service accounts | ||
|
|
||
| Service accounts can be managed as part of each project's YAML configuration. This allows creation of default service accounts used for GCE instances, in firewall rules, or for application-level credentials without resorting to a separate Terraform configuration. | ||
|
|
||
| Each service account is represented by one key and a set of optional key/value pairs in the `service_accounts` top-level YAML map, like in this example: | ||
|
|
||
| ```yaml | ||
| service_accounts: | ||
| be-0: {} | ||
| fe-1: | ||
| display_name: GCE frontend service account. | ||
| iam_project_roles: | ||
| - roles/storage.objectViewer | ||
| ``` | ||
| Both the `display_name` and `iam_project_roles` attributes are optional. | ||
|
|
||
| ### Billing budgets | ||
|
|
||
| The project factory integrates the billing budgets factory exposed by the `[`billing-account`](../billing-account/) module, and adds support for easy referencing budgets in project files. | ||
|
|
||
| To enable support for billing budgets, set the billing account id, optional notification channels, and the data folder for budgets in the `factories_config.budgets` variable, then create billing budgets using YAML definitions following the format described in the `billing-account` module. | ||
|
|
||
| Once budgets are defined, they can be referenced in a project file using their file name: | ||
|
|
||
| ```yaml | ||
| billing_account: 012345-67890A-BCDEF0 | ||
| labels: | ||
| app: app-1 | ||
| team: foo | ||
| parent: folders/12345678 | ||
| services: | ||
| - container.googleapis.com | ||
| - storage.googleapis.com | ||
| billing_budgets: | ||
| - test-100 | ||
| ``` | ||
|
|
||
| The example below shows how to use the billing budgets factory. | ||
|
|
||
| ## Example | ||
|
|
||
|
|
@@ -35,7 +86,7 @@ module "project-factory" { | |
| source = "./fabric/modules/project-factory" | ||
| # use a default billing account if none is specified via yaml | ||
| data_defaults = { | ||
| billing_account = "012345-67890A-ABCDEF" | ||
| billing_account = var.billing_account_id | ||
| } | ||
| # make sure the environment label and stackdriver service are always added | ||
| data_merges = { | ||
|
|
@@ -54,12 +105,28 @@ module "project-factory" { | |
| prefix = "test-pf" | ||
| } | ||
| # location where the yaml files are read from | ||
| factory_data_path = "data" | ||
| factories_config = { | ||
| budgets = { | ||
| billing_account = var.billing_account_id | ||
| budgets_data_path = "data/budgets" | ||
| notification_channels = { | ||
| billing-default = { | ||
| project_id = "foo-billing-audit" | ||
| type = "email" | ||
| labels = { | ||
| email_address = "[email protected]" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| projects_data_path = "data/projects" | ||
| } | ||
| } | ||
| # tftest modules=7 resources=33 files=prj-app-1,prj-app-2,prj-app-3 inventory=example.yaml | ||
| # tftest modules=8 resources=35 files=prj-app-1,prj-app-2,prj-app-3,budget-test-100 inventory=example.yaml | ||
| ``` | ||
|
|
||
| ```yaml | ||
| # project app-1 | ||
| billing_account: 012345-67890A-BCDEF0 | ||
| labels: | ||
| app: app-1 | ||
|
|
@@ -78,11 +145,13 @@ service_accounts: | |
| - roles/monitoring.metricWriter | ||
| app-1-fe: | ||
| display_name: "Test app 1 frontend." | ||
|
|
||
| # tftest-file id=prj-app-1 path=data/prj-app-1.yaml | ||
| billing_budgets: | ||
| - test-100 | ||
| # tftest-file id=prj-app-1 path=data/projects/prj-app-1.yaml | ||
| ``` | ||
|
|
||
| ```yaml | ||
| # project app-2 | ||
| labels: | ||
| app: app-2 | ||
| team: foo | ||
|
|
@@ -115,24 +184,45 @@ shared_vpc_service_config: | |
| europe-west1/prod-default-ew1: | ||
| - group:[email protected] | ||
| # tftest-file id=prj-app-2 path=data/prj-app-2.yaml | ||
| # tftest-file id=prj-app-2 path=data/projects/prj-app-2.yaml | ||
| ``` | ||
|
|
||
| ```yaml | ||
| # project app-3 | ||
| parent: folders/12345678 | ||
| services: | ||
| - run.googleapis.com | ||
| - storage.googleapis.com | ||
| # tftest-file id=prj-app-3 path=data/prj-app-3.yaml | ||
| # tftest-file id=prj-app-3 path=data/projects/prj-app-3.yaml | ||
| ``` | ||
|
|
||
| ```yaml | ||
| # billing budget test-100 | ||
| display_name: 100 dollars in current spend | ||
| amount: | ||
| units: 100 | ||
| filter: | ||
| period: | ||
| calendar: MONTH | ||
| resource_ancestors: | ||
| - folders/1234567890 | ||
| threshold_rules: | ||
| - percent: 0.5 | ||
| - percent: 0.75 | ||
| update_rules: | ||
| default: | ||
| disable_default_iam_recipients: true | ||
| monitoring_notification_channels: | ||
| - billing-default | ||
| # tftest-file id=budget-test-100 path=data/budgets/test-100.yaml | ||
| ``` | ||
| <!-- BEGIN TFDOC --> | ||
| ## Variables | ||
|
|
||
| | name | description | type | required | default | | ||
| |---|---|:---:|:---:|:---:| | ||
| | [factory_data_path](variables.tf#L91) | Path to folder with YAML project description data files. | <code>string</code> | ✓ | | | ||
| | [factories_config](variables.tf#L91) | Path to folder with YAML resource description data files. | <code title="object({ projects_data_path = string budgets = optional(object({ billing_account = string budgets_data_path = string notification_channels = optional(map(any), {}) })) })">object({…})</code> | ✓ | | | ||
| | [data_defaults](variables.tf#L17) | Optional default values used when corresponding project data from files are missing. | <code title="object({ billing_account = optional(string) contacts = optional(map(list(string)), {}) labels = optional(map(string), {}) metric_scopes = optional(list(string), []) parent = optional(string) prefix = optional(string) service_encryption_key_ids = optional(map(list(string)), {}) service_perimeter_bridges = optional(list(string), []) service_perimeter_standard = optional(string) services = optional(list(string), []) shared_vpc_service_config = optional(object({ host_project = string network_users = optional(list(string), []) service_identity_iam = optional(map(list(string)), {}) service_identity_subnet_iam = optional(map(list(string)), {}) service_iam_grants = optional(list(string), []) network_subnet_users = optional(map(list(string)), {}) }), { host_project = null }) tag_bindings = optional(map(string), {}) service_accounts = optional(map(object({ display_name = optional(string, "Terraform-managed.") iam_project_roles = optional(list(string)) })), {}) })">object({…})</code> | | <code>{}</code> | | ||
| | [data_merges](variables.tf#L49) | Optional values that will be merged with corresponding data from files. Combines with `data_defaults`, file data, and `data_overrides`. | <code title="object({ contacts = optional(map(list(string)), {}) labels = optional(map(string), {}) metric_scopes = optional(list(string), []) service_encryption_key_ids = optional(map(list(string)), {}) service_perimeter_bridges = optional(list(string), []) services = optional(list(string), []) tag_bindings = optional(map(string), {}) service_accounts = optional(map(object({ display_name = optional(string, "Terraform-managed.") iam_project_roles = optional(list(string)) })), {}) })">object({…})</code> | | <code>{}</code> | | ||
| | [data_overrides](variables.tf#L69) | Optional values that override corresponding data from files. Takes precedence over file data and `data_defaults`. | <code title="object({ billing_account = optional(string) contacts = optional(map(list(string))) parent = optional(string) prefix = optional(string) service_encryption_key_ids = optional(map(list(string))) service_perimeter_bridges = optional(list(string)) service_perimeter_standard = optional(string) tag_bindings = optional(map(string)) services = optional(list(string)) service_accounts = optional(map(object({ display_name = optional(string, "Terraform-managed.") iam_project_roles = optional(list(string)) }))) })">object({…})</code> | | <code>{}</code> | | ||
|
|
@@ -144,7 +234,6 @@ services: | |
| | [projects](outputs.tf#L17) | Project module outputs. | | | ||
| | [service_accounts](outputs.tf#L22) | Service account emails. | | | ||
| <!-- END TFDOC --> | ||
|
|
||
| ## Tests | ||
|
|
||
| These tests validate fixes to the project factory. | ||
|
|
@@ -166,7 +255,9 @@ module "project-factory" { | |
| data_overrides = { | ||
| prefix = "foo" | ||
| } | ||
| factory_data_path = "data" | ||
| factories_config = { | ||
| projects_data_path = "data/projects" | ||
| } | ||
| } | ||
| # tftest modules=4 resources=14 files=test-0,test-1,test-2 | ||
| ``` | ||
|
|
@@ -177,21 +268,21 @@ services: | |
| - iam.googleapis.com | ||
| - contactcenteraiplatform.googleapis.com | ||
| - container.googleapis.com | ||
| # tftest-file id=test-0 path=data/test-0.yaml | ||
| # tftest-file id=test-0 path=data/projects/test-0.yaml | ||
| ``` | ||
|
|
||
| ```yaml | ||
| parent: folders/1234567890 | ||
| services: | ||
| - iam.googleapis.com | ||
| - contactcenteraiplatform.googleapis.com | ||
| # tftest-file id=test-1 path=data/test-1.yaml | ||
| # tftest-file id=test-1 path=data/projects/test-1.yaml | ||
| ``` | ||
|
|
||
| ```yaml | ||
| parent: folders/1234567890 | ||
| services: | ||
| - iam.googleapis.com | ||
| - storage.googleapis.com | ||
| # tftest-file id=test-2 path=data/test-2.yaml | ||
| # tftest-file id=test-2 path=data/projects/test-2.yaml | ||
| ``` | ||
Oops, something went wrong.