Business Units org-bootstrap example

organization-bootstrap/business-units/README.md

@@ -0,0 +1,69 @@
+# Business-units based organizational sample
+
+This sample creates an organizational layout with two folder levels, where the first level is usually mapped to one business unit or team (infra, data, analytics) and the second level represents enviroments (prod, test). It also sets up all prerequisites for automation (GCS state buckets, service accounts, etc.), and the correct roles on those to enforce separation of duties at the environment level.
+
+This layout is well suited for medium-sized infrastructures managed by different sets of teams, and especially where the foundational infrastructure needs to be managed centrally, as the top-level automation service accounts for each environment allow cross-team management of the base resources (projects, IAM, etc.).
+
+![High-level diagram](diagram.png "High-level diagram")
+
+Refer to the [section-level README](../README.md) for general considerations about this type of samples, and usage instructions.
+
+## Managed resources and services
+
+This sample creates several distinct groups of resources:
+
+- one top-level folder per business unit/team
+- one top-level folder for shared services
+- one second-level folder for each environment in all the business unit top-level folders
+- one project in the shared folder to hold Terraform-related resources
+- one project in the shared folder to set up and host centralized audit log exports
+- one project in the shared folder to hold services used across environments like GCS, GCR, KMS, Cloud Build, etc.
+
+The number of resources in this sample is kept to a minimum so as to make it generally applicable, more resources can be easily added by leveraging the full array of [Cloud Foundation Toolkit modules](https://github.com/terraform-google-modules), especially in the shared services project.
+
+## Shared services
+
+This sample uses a top-level folder to encapsulate projects that host resources that are not specific to a single environment. If no shared services are needed,the Terraform and audit modules can be easily attached to the root node, and the shared services folder and project removed from `main.tf`.
+
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|:----:|:-----:|:-----:|
+| audit\_viewers | Audit project viewers, in IAM format. | list | `<list>` | no |
+| billing\_account\_id | Billing account id used as default for new projects. | string | n/a | yes |
+| business\_unit\_1\_name | Business unit 1 short name. | string | n/a | yes |
+| business\_unit\_2\_name | Business unit 2 short name. | string | n/a | yes |
+| business\_unit\_3\_name | Business unit 3 short name. | string | n/a | yes |
+| environments | Environment short names. | list(string) | n/a | yes |
+| gcs\_location | GCS bucket location. | string | `"EU"` | no |
+| generate\_service\_account\_keys | Generate and store service account keys in the state file. | string | `"false"` | no |
+| organization\_id | Organization id. | string | n/a | yes |
+| prefix | Prefix used for resources that need unique names. | string | n/a | yes |
+| project\_services | Service APIs enabled by default in new projects. | list | `<list>` | no |
+| root\_node | Root node for the new hierarchy, either 'organizations/org_id' or 'folders/folder_id'. | string | n/a | yes |
+| shared\_bindings\_members | List of comma-delimited IAM-format members for the additional shared project bindings. | list | `<list>` | no |
+| shared\_bindings\_roles | List of roles for additional shared project bindings. | list | `<list>` | no |
+| terraform\_owners | Terraform project owners, in IAM format. | list | `<list>` | no |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| audit\_logs\_bq\_dataset | Bigquery dataset for the audit logs export. |
+| audit\_logs\_project | Project that holds the audit logs export resources. |
+| bootstrap\_tf\_gcs\_bucket | GCS bucket used for the bootstrap Terraform state. |
+| business\_unit\_1\_environment\_folders\_ids | Business unit 1 environment folders. |
+| business\_unit\_1\_folder\_id | Business unit 1 top-level folder ID. |
+| business\_unit\_2\_environment\_folders\_ids | Business unit 2 environment folders. |
+| business\_unit\_2\_folder\_id | Business unit 2 top-level folder ID. |
+| business\_unit\_3\_environment\_folders\_ids | Business unit 3 environment folders. |
+| business\_unit\_3\_folder\_id | Business unit 3 top-level folder ID. |
+| environment\_service\_account\_keys | Service account keys used to run each environment Terraform modules. |
+| environment\_service\_accounts | Service accounts used to run each environment Terraform modules. |
+| environment\_tf\_gcs\_buckets | GCS buckets used for each environment Terraform state. |
+| shared\_folder\_id | Shared folder ID. |
+| shared\_resources\_project | Project that holdes resources shared across business units. |
+| terraform\_project | Project that holds the base Terraform resources. |
+
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

organization-bootstrap/business-units/backend.tf.sample

@@ -0,0 +1,22 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+terraform {
+  backend "gcs" {
+    # once initial apply has completed, copy this file to `backend.tf` then
+    # set the `bucket` value to the `bootstrap_tf_gcs_bucket` output, then
+    # run apply again to transfer state
+    bucket = ""
+  }
+}

organization-bootstrap/business-units/main.tf

@@ -0,0 +1,195 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# TODO(averbukh): simplify log-sink parameters once https://github.com/terraform-google-modules/terraform-google-log-export/issues/28 is done. 
+
+locals {
+  parent_numeric_id             = element(split("/", var.root_node), 1)
+  log_sink_parent_resource_type = element(split("/", var.root_node), 0) == "organizations" ? "organization" : "folder"
+  log_sink_name                 = element(split("/", var.root_node), 0) == "organizations" ? "logs-audit-org-${local.parent_numeric_id}" : "logs-audit-folder-${local.parent_numeric_id}"
+}
+
+###############################################################################
+#                        Shared resources folder                              #
+###############################################################################
+
+module "shared-folder" {
+  source  = "terraform-google-modules/folders/google"
+  version = "2.0.0"
+  parent  = var.root_node
+  names   = ["shared"]
+}
+
+###############################################################################
+#                        Terraform top-level resources                        #
+###############################################################################
+
+# Terraform project
+
+module "project-tf" {
+  source          = "terraform-google-modules/project-factory/google//modules/fabric-project"
+  version         = "3.2.0"
+  parent          = module.shared-folder.id
+  billing_account = var.billing_account_id
+  prefix          = var.prefix
+  name            = "terraform"
+  lien_reason     = "terraform"
+  owners          = var.terraform_owners
+  activate_apis   = var.project_services
+}
+
+# Per environment service accounts
+
+module "service-accounts-tf-environments" {
+  source             = "terraform-google-modules/service-accounts/google"
+  version            = "2.0.0"
+  project_id         = module.project-tf.project_id
+  org_id             = var.organization_id
+  billing_account_id = var.billing_account_id
+  prefix             = var.prefix
+  names              = var.environments
+  grant_billing_role = true
+  generate_keys      = var.generate_service_account_keys
+}
+
+# Bootstrap Terraform state GCS bucket
+
+module "gcs-tf-bootstrap" {
+  source     = "terraform-google-modules/cloud-storage/google"
+  version    = "1.0.0"
+  project_id = module.project-tf.project_id
+  prefix     = "${var.prefix}-tf"
+  names      = ["tf-bootstrap"]
+  location   = var.gcs_location
+}
+
+# Per environment Terraform state GCS buckets
+
+module "gcs-tf-environments" {
+  source          = "terraform-google-modules/cloud-storage/google"
+  version         = "1.0.0"
+  project_id      = module.project-tf.project_id
+  prefix          = "${var.prefix}-tf"
+  names           = var.environments
+  location        = var.gcs_location
+  set_admin_roles = true
+  bucket_admins = zipmap(
+    var.environments,
+    module.service-accounts-tf-environments.iam_emails_list
+  )
+}
+
+###############################################################################
+#                              Business units                                 #
+###############################################################################
+
+# Business unit 1
+
+module "business-unit-1-folders" {
+  source = "./modules/business-unit-folders"
+
+  business_unit_folder_name = var.business_unit_1_name
+  environments              = var.environments
+  per_folder_admins         = module.service-accounts-tf-environments.iam_emails_list
+  root_node                 = var.root_node
+
+}
+
+# Business unit 2
+
+module "business-unit-2-folders" {
+  source = "./modules/business-unit-folders"
+
+  business_unit_folder_name = var.business_unit_2_name
+  environments              = var.environments
+  per_folder_admins         = module.service-accounts-tf-environments.iam_emails_list
+  root_node                 = var.root_node
+
+}
+
+# Business unit 3
+
+module "business-unit-3-folders" {
+  source = "./modules/business-unit-folders"
+
+  business_unit_folder_name = var.business_unit_3_name
+  environments              = var.environments
+  per_folder_admins         = module.service-accounts-tf-environments.iam_emails_list
+  root_node                 = var.root_node
+
+}
+
+###############################################################################
+#                              Audit log exports                              #
+###############################################################################
+
+# Audit logs project
+
+module "project-audit" {
+  source          = "terraform-google-modules/project-factory/google//modules/fabric-project"
+  version         = "3.2.0"
+  parent          = module.shared-folder.id
+  billing_account = var.billing_account_id
+  prefix          = var.prefix
+  name            = "audit"
+  lien_reason     = "audit"
+  activate_apis   = var.project_services
+  viewers         = var.audit_viewers
+}
+
+# Audit logs destination on BigQuery
+
+module "bq-audit-export" {
+  source                   = "terraform-google-modules/log-export/google//modules/bigquery"
+  version                  = "3.0.0"
+  project_id               = module.project-audit.project_id
+  dataset_name             = "${replace(local.log_sink_name, "-", "_")}"
+  log_sink_writer_identity = module.log-sink-audit.writer_identity
+}
+
+# Audit log sink for root node
+
+module "log-sink-audit" {
+  source                 = "terraform-google-modules/log-export/google"
+  version                = "3.0.0"
+  filter                 = "logName: \"/logs/cloudaudit.googleapis.com%2Factivity\" OR logName: \"/logs/cloudaudit.googleapis.com%2Fsystem_event\""
+  log_sink_name          = local.log_sink_name
+  parent_resource_type   = local.log_sink_parent_resource_type
+  parent_resource_id     = local.parent_numeric_id
+  include_children       = "true"
+  unique_writer_identity = "true"
+  destination_uri        = "${module.bq-audit-export.destination_uri}"
+}
+
+###############################################################################
+#                    Shared resources (GCR, GCS, KMS, etc.)                   #
+###############################################################################
+
+# Shared resources project
+
+module "project-shared-resources" {
+  source                 = "terraform-google-modules/project-factory/google//modules/fabric-project"
+  version                = "3.2.0"
+  parent                 = module.shared-folder.id
+  billing_account        = var.billing_account_id
+  prefix                 = var.prefix
+  name                   = "shared"
+  lien_reason            = "shared"
+  activate_apis          = var.project_services
+  extra_bindings_roles   = var.shared_bindings_roles
+  extra_bindings_members = var.shared_bindings_members
+}
+
+# Add further modules here for resources that are common to all business units
+# like GCS buckets (used to hold shared assets), Container Registry, KMS, etc.

organization-bootstrap/business-units/modules/business-unit-folders/README.md

@@ -0,0 +1,35 @@
+# Two-levels folders tree
+
+This module creates one top-level folder and set of second-level folders, which meant to represent a business unit with set of enviroments underneeth. It also creates per environment GCS bucket and ServiceAccount in a project specified by `tf_project_id` variable. 
+
+This set of Terraform files is usually applied manually by an org-level administrator as a first step, and then reapplied only when a new environment needs to be created or an existing one removed, and serves different purposes:
+
+## Managed resources and services
+
+This sample creates several distinct groups of resources:
+
+- one top level folder
+- set of second level folders
+- set of GCS buckets, which meant to be used for storing terraform state. 
+- set of service accounts and keys, which meant to be used for automating underneeth infrastructure with terraform. 
+
+The number of resources in this sample is kept to a minimum so as to make it generally applicable, more resources can be easily added by leveraging the full array of [Cloud Foundation Toolkit modules](https://github.com/terraform-google-modules), especially in the shared services project.
+
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|:----:|:-----:|:-----:|
+| business\_unit\_folder\_name | Business Unit Folder name. | string | n/a | yes |
+| environments | Environment short names. | list(string) | n/a | yes |
+| per\_folder\_admins | List of IAM-style members per folder who will get extended permissions. | list(string) | `<list>` | no |
+| root\_node | Root node for the new hierarchy, either 'organizations/org_id' or 'folders/folder_id'. | string | n/a | yes |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| business\_unit\_folder\_id | Business Unit Folder ID. |
+| environment\_folders\_ids | Environment folders IDs. |
+
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

organization-bootstrap/business-units/modules/business-unit-folders/main.tf

@@ -0,0 +1,44 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+###############################################################################
+#                              Business Unit Folder                           #
+###############################################################################
+
+module "business-unit-folder" {
+  source  = "terraform-google-modules/folders/google"
+  version = "2.0.0"
+  parent  = var.root_node
+  names   = [var.business_unit_folder_name]
+}
+
+###############################################################################
+#                              Environment Folders                            #
+###############################################################################
+
+module "environment-folders" {
+  source            = "terraform-google-modules/folders/google"
+  version           = "2.0.0"
+  parent            = module.business-unit-folder.id
+  names             = var.environments
+  set_roles         = true
+  per_folder_admins = var.per_folder_admins
+  folder_admin_roles = [
+    "roles/resourcemanager.folderViewer",
+    "roles/resourcemanager.projectCreator",
+    "roles/owner",
+    "roles/compute.networkAdmin",
+    "roles/compute.xpnAdmin"
+  ]
+}

organization-bootstrap/business-units/modules/business-unit-folders/outputs.tf

@@ -0,0 +1,26 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+output "business_unit_folder_id" {
+  description = "Business Unit Folder ID."
+  value       = module.business-unit-folder.id
+}
+
+output "environment_folders_ids" {
+  description = "Environment folders IDs."
+  value       = module.environment-folders.ids
+}
+
+# Add further outputs here for the additional modules that manage shared
+# resources, like GCR, GCS buckets, KMS, etc.