Averbuks org business units

.gitignore

@@ -0,0 +1,8 @@
+**/.terraform
+**/terraform.tfstate*
+**/terraform.tfvars
+.idea
+.vscode
+backend-config.hcl
+credentials.json
+key.json

organization-bootstrap/business-units/README.md

@@ -0,0 +1,102 @@
+# Business-units based organizational sample
+
+This sample creates an organizational layout with two folder levels, where the first level is usually mapped to one business unit (infra, data, analythics) and the second level represents enviroments (prod, test). It also sets up all prerequisites for automation (GCS state buckets, service accounts, etc.), and the correct roles on those to enforce separation of duties at the environment level.
+
+This layout is well suited for small and medium-sized infrastructures managed by a smal set of teams, where the complexity in application resource ownership and access roles is mostly dealt with at the project level, and/or in the individual services (GKE, Cloud SQL, etc.). Its simplicity also makes it a good starting point for more complex or specialized layouts.
+
+This layout is well suited for medium-sized infrastructures managed by different sets of teams grouped to different business units, where the complexity in application resource ownership and access roles is mostly dealt with at the project level, and/or in the individual services (GKE, Cloud SQL, etc.). 
+
+![High-level diagram](diagram.png "High-level diagram")
+
+This set of Terraform files is usually applied manually by an org-level administrator as a first step, and then reapplied only when a new business-unit or environment needs to be created or an existing one removed, and serves different purposes:
+
+- automating and parameterizing creation of the organizational layout
+- automating creation of the base resources needed for Terraform automation, obviating the need for external scripts or hand-coded commands
+- anticipating the requirement of organizational-level roles for specific resources (eg Shared VPC), by granting them to the service accounts used for environment automation
+- enforcing separation of duties by using separate sets of automation resources (GCS, service accounts) for each environment, and only granting roles scoped to the environment's folder
+
+## Managed resources and services
+
+This sample creates several distinct groups of resources:
+
+- one folder per business unit
+- two second-level folders (test, prod) for every business unit
+- one top-level project to hold Terraform-related resources
+- one top-level project to set up and host centralized audit log exports (optional).
+- one top-level project to hold services used across environments like GCS, GCR, KMS, Cloud Build, etc. (optional)
+
+The number of resources in this sample is kept to a minimum so as to make it generally applicable, more resources can be easily added by leveraging the full array of [Cloud Foundation Toolkit modules](https://github.com/terraform-google-modules), especially in the shared services project.
+
+## Operational considerations
+
+As mentioned above this root module is meant to be run infrequently, only when an environment or a shared service needs to be added or changed, so the advantages of automating it in a CI pipeline are very limited.
+
+### IAM roles
+
+Regardless of how it's run, the credentials used need very specific roles on the root node, plus additional roles at the organization level if Shared VPC usage is anticipated in environments:
+
+- Billing Account Administrator on the billing account or organization
+- Folder Administrator
+- Logging Administrator on the root folder or organization
+- Project Creator
+- Organization Administrator, if Shared VPC roles need to be granted
+
+### State
+
+TODO: describe the state switch that needs to be done after first apply
+
+### Things to be aware of
+
+TODO: describe potential issues with multiple resources, and the upcoming `foreach` fix
+TODO: describe how `prefix` can be used to enforce naming
+TODO: describe xpn roles
+
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|:----:|:-----:|:-----:|
+| audit\_viewers | Audit project viewers, in IAM format. | list | `<list>` | no |
+| billing\_account\_id | Billing account id used as default for new projects. | string | n/a | yes |
+| business\_unit\_1\_envs | Business unit 1 environments short names. | list(string) | `<list>` | no |
+| business\_unit\_1\_name | Business unit 1 short name. | string | n/a | yes |
+| business\_unit\_2\_envs | Business unit 2 environments short names. | list(string) | `<list>` | no |
+| business\_unit\_2\_name | Business unit 2 short name. | string | n/a | yes |
+| business\_unit\_3\_envs | Business unit 3 environments short names. | list(string) | `<list>` | no |
+| business\_unit\_3\_name | Business unit 3 short name. | string | n/a | yes |
+| gcs\_location | GCS bucket location. | string | `"EU"` | no |
+| generate\_service\_account\_keys | Generate and store service account keys in the state file. | string | `"false"` | no |
+| organization\_id | Organization id. | string | n/a | yes |
+| prefix | Prefix used for resources that need unique names. | string | n/a | yes |
+| project\_services | Service APIs enabled by default in new projects. | list | `<list>` | no |
+| root\_node | Root node for the new hierarchy, either 'organizations/org_id' or 'folders/folder_id'. | string | n/a | yes |
+| shared\_bindings\_members | List of comma-delimited IAM-format members for the additional shared project bindings. | list | `<list>` | no |
+| shared\_bindings\_roles | List of roles for additional shared project bindings. | list | `<list>` | no |
+| terraform\_owners | Terraform project owners, in IAM format. | list | `<list>` | no |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| audit\_logs\_bq\_dataset | Bigquery dataset for the audit logs export. |
+| audit\_logs\_project | Project that holds the audit logs export resources. |
+| bootstrap\_tf\_gcs\_bucket | GCS bucket used for the bootstrap Terraform state. |
+| business\_unit\_1\_envs\_folders\_ids | Business unit 1 environment folders. |
+| business\_unit\_1\_envs\_service\_account\_keys | Service account keys used to run each environment Terraform modules for business unit 1. |
+| business\_unit\_1\_envs\_service\_accounts | Service accounts used to run each environment Terraform modules for business unit 1. |
+| business\_unit\_1\_envs\_tf\_gcs\_buckets | GCS buckets used for each environment Terraform state for business unit 1. |
+| business\_unit\_1\_top\_level\_folder\_id | Business unit 1 top-level folder. |
+| business\_unit\_2\_envs\_folders\_ids | Business unit 2 environment folders. |
+| business\_unit\_2\_envs\_service\_account\_keys | Service account keys used to run each environment Terraform modules for business unit 2. |
+| business\_unit\_2\_envs\_service\_accounts | Service accounts used to run each environment Terraform modules for business unit 2. |
+| business\_unit\_2\_envs\_tf\_gcs\_buckets | GCS buckets used for each environment Terraform state for business unit 2. |
+| business\_unit\_2\_top\_level\_folder\_id | Business unit 2 top-level folder. |
+| business\_unit\_3\_envs\_folders\_ids | Business unit 3 environment folders. |
+| business\_unit\_3\_envs\_service\_account\_keys | Service account keys used to run each environment Terraform modules for business unit 3. |
+| business\_unit\_3\_envs\_service\_accounts | Service accounts used to run each environment Terraform modules for business unit 3. |
+| business\_unit\_3\_envs\_tf\_gcs\_buckets | GCS buckets used for each environment Terraform state for business unit 3. |
+| business\_unit\_3\_top\_level\_folder\_id | Business unit 3 top-level folder. |
+| shared\_resources\_project | Project that holdes resources shared across business units. |
+| terraform\_project | Project that holds the base Terraform resources. |
+
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->

organization-bootstrap/business-units/backend.tf

@@ -0,0 +1,21 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# comment before initial run
+
+terraform {
+  backend "gcs" {
+    bucket = ""
+  }
+}

organization-bootstrap/business-units/main.tf

@@ -0,0 +1,162 @@
+# Copyright 2019 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+###############################################################################
+#                        Terraform top-level resources                        #
+###############################################################################
+
+# Terraform project
+
+module "project-tf" {
+  source          = "terraform-google-modules/project-factory/google//modules/fabric-project"
+  version         = "3.2.0"
+  parent          = var.root_node
+  billing_account = var.billing_account_id
+  prefix          = var.prefix
+  name            = "terraform"
+  lien_reason     = "terraform"
+  owners          = var.terraform_owners
+  activate_apis   = var.project_services
+}
+
+# Bootstrap Terraform state GCS bucket
+
+module "gcs-tf-bootstrap" {
+  source     = "terraform-google-modules/cloud-storage/google"
+  version    = "1.0.0"
+  project_id = module.project-tf.project_id
+  prefix     = "${var.prefix}-tf"
+  names      = ["tf-bootstrap"]
+  location   = var.gcs_location
+}
+
+###############################################################################
+#                              Business units                                 #
+###############################################################################
+
+# Business unit 1
+
+module "business-unit-1-folders-tree" {
+  source = "./modules/business-unit-folders-tree"
+
+  billing_account_id            = var.billing_account_id
+  tf_project_id                 = module.project-tf.project_id
+  top_level_folder_name         = var.business_unit_1_name
+  second_level_folders_names    = var.business_unit_1_envs
+  generate_service_account_keys = var.generate_service_account_keys
+  gcs_location                  = var.gcs_location
+  organization_id               = var.organization_id
+  prefix                        = var.prefix
+  root_node                     = var.root_node
+
+}
+
+# Business unit 2
+
+module "business-unit-2-folders-tree" {
+  source = "./modules/business-unit-folders-tree"
+
+  billing_account_id            = var.billing_account_id
+  tf_project_id                 = module.project-tf.project_id
+  top_level_folder_name         = var.business_unit_2_name
+  second_level_folders_names    = var.business_unit_2_envs
+  generate_service_account_keys = var.generate_service_account_keys
+  gcs_location                  = var.gcs_location
+  organization_id               = var.organization_id
+  prefix                        = var.prefix
+  root_node                     = var.root_node
+
+}
+
+# Business unit 3
+
+module "business-unit-3-folders-tree" {
+  source = "./modules/business-unit-folders-tree"
+
+  billing_account_id            = var.billing_account_id
+  tf_project_id                 = module.project-tf.project_id
+  top_level_folder_name         = var.business_unit_3_name
+  second_level_folders_names    = var.business_unit_3_envs
+  generate_service_account_keys = var.generate_service_account_keys
+  gcs_location                  = var.gcs_location
+  organization_id               = var.organization_id
+  prefix                        = var.prefix
+  root_node                     = var.root_node
+
+}
+
+###############################################################################
+#                              Audit log exports                              #
+###############################################################################
+
+# Audit logs project
+
+module "project-audit" {
+  source          = "terraform-google-modules/project-factory/google//modules/fabric-project"
+  version         = "3.2.0"
+  parent          = var.root_node
+  billing_account = var.billing_account_id
+  prefix          = var.prefix
+  name            = "audit"
+  lien_reason     = "audit"
+  activate_apis   = var.project_services
+  viewers         = var.audit_viewers
+}
+
+# Audit logs destination on BigQuery
+
+module "bq-audit-export" {
+  source                   = "terraform-google-modules/log-export/google//modules/bigquery"
+  version                  = "3.0.0"
+  project_id               = module.project-audit.project_id
+  dataset_name             = "logs_audit_${replace(var.business_unit_1_name, "-", "_")}"
+  log_sink_writer_identity = module.log-sink-audit.writer_identity
+}
+
+# Audit log sink for business unit 1
+# set the organization as parent to export audit logs for all business units
+
+module "log-sink-audit" {
+  source                 = "terraform-google-modules/log-export/google"
+  version                = "3.0.0"
+  filter                 = "logName: \"/logs/cloudaudit.googleapis.com%2Factivity\" OR logName: \"/logs/cloudaudit.googleapis.com%2Fsystem_event\""
+  log_sink_name          = "logs-audit-${var.business_unit_1_name}"
+  parent_resource_type   = "folder"
+  parent_resource_id     = split("/", module.business-unit-1-folders-tree.top_level_folder_id)[1]
+  include_children       = "true"
+  unique_writer_identity = "true"
+  destination_uri        = "${module.bq-audit-export.destination_uri}"
+}
+
+###############################################################################
+#                    Shared resources (GCR, GCS, KMS, etc.)                   #
+###############################################################################
+
+# Shared resources project
+
+module "project-shared-resources" {
+  source                 = "terraform-google-modules/project-factory/google//modules/fabric-project"
+  version                = "3.2.0"
+  parent                 = var.root_node
+  billing_account        = var.billing_account_id
+  prefix                 = var.prefix
+  name                   = "shared"
+  lien_reason            = "shared"
+  activate_apis          = var.project_services
+  extra_bindings_roles   = var.shared_bindings_roles
+  extra_bindings_members = var.shared_bindings_members
+}
+
+# Add further modules here for resources that are common to all business units
+# like GCS buckets (used to hold shared assets), Container Registry, KMS, etc.

organization-bootstrap/business-units/modules/business-unit-folders-tree/README.md

@@ -0,0 +1,43 @@
+# Two-levels folders tree
+
+This module creates one top-level folder and set of second-level folders, which meant to represent a business unit with set of enviroments underneeth. It also creates per environment GCS bucket and ServiceAccount in a project specified by `tf_project_id` variable. 
+
+This set of Terraform files is usually applied manually by an org-level administrator as a first step, and then reapplied only when a new environment needs to be created or an existing one removed, and serves different purposes:
+
+## Managed resources and services
+
+This sample creates several distinct groups of resources:
+
+- one top level folder
+- set of second level folders
+- set of GCS buckets, which meant to be used for storing terraform state. 
+- set of service accounts and keys, which meant to be used for automating underneeth infrastructure with terraform. 
+
+The number of resources in this sample is kept to a minimum so as to make it generally applicable, more resources can be easily added by leveraging the full array of [Cloud Foundation Toolkit modules](https://github.com/terraform-google-modules), especially in the shared services project.
+
+<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
+## Inputs
+
+| Name | Description | Type | Default | Required |
+|------|-------------|:----:|:-----:|:-----:|
+| billing\_account\_id | Billing account id used as default for new projects. | string | n/a | yes |
+| gcs\_location | GCS bucket location. | string | `"EU"` | no |
+| generate\_service\_account\_keys | Generate and store service account keys in the state file. | string | `"false"` | no |
+| organization\_id | Organization id. | string | n/a | yes |
+| prefix | Prefix used for resources that need unique names. | string | n/a | yes |
+| root\_node | Root node for the new hierarchy, either 'organizations/org_id' or 'folders/folder_id'. | string | n/a | yes |
+| second\_level\_folders\_names | Second level folders names. | list(string) | n/a | yes |
+| tf\_project\_id | Id of a project where service accounts and terraform state buckets should be created | string | n/a | yes |
+| top\_level\_folder\_name | Top level folder name. | string | n/a | yes |
+
+## Outputs
+
+| Name | Description |
+|------|-------------|
+| second\_level\_folders\_ids | Second-level folders IDs. |
+| second\_level\_folders\_service\_account\_emails | Service accounts used to run each econd-level folder Terraform modules. |
+| second\_level\_folders\_service\_account\_keys | Service account keys used to run each second-level folder Terraform modules. |
+| second\_level\_folders\_tf\_gcs\_bucket\_names | GCS buckets used for each second-level folder Terraform state. |
+| top\_level\_folder\_id | Top-level folder ID. |
+
+<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->