Skip to content

Commit

Permalink
business-units foundation example (#52)
Browse files Browse the repository at this point in the history
* Added folder-units module.

* Business units example update (WIP)

* Update all BU modules to internal ones

* Refactoring business-units example, add billing and org IAM handling

* update projects tests for new iam additive naming

* update project README for new iam additive naming

* streamline bu example and module (#53)

Co-authored-by: Ludovico Magnocavallo <[email protected]>
  • Loading branch information
averbuks and ludoo authored Mar 23, 2020
1 parent 9c1eb24 commit 708ca97
Show file tree
Hide file tree
Showing 28 changed files with 744 additions and 524 deletions.
36 changes: 13 additions & 23 deletions foundations/business-units/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

This sample creates an organizational layout with two folder levels, where the first level is usually mapped to one business unit or team (infra, data, analytics) and the second level represents enviroments (prod, test). It also sets up all prerequisites for automation (GCS state buckets, service accounts, etc.), and the correct roles on those to enforce separation of duties at the environment level.

This layout is well suited for medium-sized infrastructures managed by different sets of teams, and especially where the foundational infrastructure needs to be managed centrally, as the top-level automation service accounts for each environment allow cross-team management of the base resources (projects, IAM, etc.).
This layout is well suited for medium-sized infrastructures managed by different sets of teams, and in cases where the core infrastructure is managed centrally, as the top-level automation service accounts for each environment allow cross-team management of the base resources (projects, IAM, etc.).

![High-level diagram](diagram.png "High-level diagram")

Expand All @@ -19,7 +19,7 @@ This sample creates several distinct groups of resources:
- one project in the shared folder to set up and host centralized audit log exports
- one project in the shared folder to hold services used across environments like GCS, GCR, KMS, Cloud Build, etc.

The number of resources in this sample is kept to a minimum so as to make it generally applicable, more resources can be easily added by leveraging the full array of [Cloud Foundation Toolkit modules](https://github.com/terraform-google-modules), especially in the shared services project.
The number of resources in this sample is kept to a minimum so as to make it generally applicable, more resources can be easily added by leveraging other [modules from our bundle](../../modules/), or from other sources like the [CFT suite](https://github.com/terraform-google-modules).

## Shared services

Expand All @@ -31,38 +31,28 @@ This sample uses a top-level folder to encapsulate projects that host resources
| name | description | type | required | default |
|---|---|:---: |:---:|:---:|
| billing_account_id | Billing account id used as default for new projects. | <code title="">string</code> || |
| business_unit_1_name | Business unit 1 short name. | <code title="">string</code> || |
| business_unit_2_name | Business unit 2 short name. | <code title="">string</code> || |
| business_unit_3_name | Business unit 3 short name. | <code title="">string</code> || |
| environments | Environment short names. | <code title="list&#40;string&#41;">list(string)</code> || |
| organization_id | Organization id. | <code title="">string</code> || |
| prefix | Prefix used for resources that need unique names. | <code title="">string</code> || |
| root_node | Root node for the new hierarchy, either 'organizations/org_id' or 'folders/folder_id'. | <code title="">string</code> || |
| *audit_viewers* | Audit project viewers, in IAM format. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |
| *gcs_location* | GCS bucket location. | <code title="">string</code> | | <code title="">EU</code> |
| *generate_service_account_keys* | Generate and store service account keys in the state file. | <code title="">bool</code> | | <code title="">false</code> |
| *audit_filter* | Audit log filter used for the log sink. | <code title="">string</code> | | <code title="&#60;&#60;END&#10;logName: &#34;&#47;logs&#47;cloudaudit.googleapis.com&#37;2Factivity&#34;&#10;OR&#10;logName: &#34;&#47;logs&#47;cloudaudit.googleapis.com&#37;2Fsystem_event&#34;&#10;END">...</code> |
| *environments* | Environment short names. | <code title="map&#40;string&#41;">map(string)</code> | | <code title="&#123;&#10;dev &#61; &#34;Development&#34;,&#10;test &#61; &#34;Testing&#34;,&#10;prod &#61; &#34;Production&#34;&#10;&#125;">...</code> |
| *gcs_defaults* | Defaults use for the state GCS buckets. | <code title="map&#40;string&#41;">map(string)</code> | | <code title="&#123;&#10;location &#61; &#34;EU&#34;&#10;storage_class &#61; &#34;MULTI_REGIONAL&#34;&#10;&#125;">...</code> |
| *iam_audit_viewers* | Audit project viewers, in IAM format. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |
| *iam_shared_owners* | Shared services project owners, in IAM format. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |
| *iam_terraform_owners* | Terraform project owners, in IAM format. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |
| *project_services* | Service APIs enabled by default in new projects. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="&#91;&#10;&#34;resourceviews.googleapis.com&#34;,&#10;&#34;stackdriver.googleapis.com&#34;,&#10;&#93;">...</code> |
| *shared_bindings_members* | List of comma-delimited IAM-format members for the additional shared project bindings. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |
| *shared_bindings_roles* | List of roles for additional shared project bindings. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |
| *terraform_owners* | Terraform project owners, in IAM format. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |

## Outputs

| name | description | sensitive |
|---|---|:---:|
| audit_logs_bq_dataset | Bigquery dataset for the audit logs export. | |
| audit_logs_project | Project that holds the audit logs export resources. | |
| bootstrap_tf_gcs_bucket | GCS bucket used for the bootstrap Terraform state. | |
| business_unit_1_environment_folders_ids | Business unit 1 environment folders. | |
| business_unit_1_folder_id | Business unit 1 top-level folder ID. | |
| business_unit_2_environment_folders_ids | Business unit 2 environment folders. | |
| business_unit_2_folder_id | Business unit 2 top-level folder ID. | |
| business_unit_3_environment_folders_ids | Business unit 3 environment folders. | |
| business_unit_3_folder_id | Business unit 3 top-level folder ID. | |
| environment_service_account_keys | Service account keys used to run each environment Terraform modules. ||
| environment_service_accounts | Service accounts used to run each environment Terraform modules. | |
| environment_tf_gcs_buckets | GCS buckets used for each environment Terraform state. | |
| shared_folder_id | Shared folder ID. | |
| bu_bi | Business Intelligence attributes. | |
| bu_bi_sa_keys | Business Intelligence service account keys. ||
| bu_ml | Machine Learning attributes. | |
| bu_ml_sa_keys | Machine Learning service account keys. ||
| shared_folder_id | Shared folder id. | |
| shared_resources_project | Project that holdes resources shared across business units. | |
| terraform_project | Project that holds the base Terraform resources. | |
<!-- END TFDOC -->
4 changes: 2 additions & 2 deletions foundations/business-units/backend.tf.sample
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright 2019 Google LLC
# Copyright 2020 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -19,4 +19,4 @@ terraform {
# run apply again to transfer state
bucket = ""
}
}
}
258 changes: 112 additions & 146 deletions foundations/business-units/main.tf
Original file line number Diff line number Diff line change
@@ -1,130 +1,86 @@
# Copyright 2019 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# TODO(averbukh): simplify log-sink parameters once https://github.com/terraform-google-modules/terraform-google-log-export/issues/28 is done.

locals {
parent_numeric_id = element(split("/", var.root_node), 1)
log_sink_parent_resource_type = element(split("/", var.root_node), 0) == "organizations" ? "organization" : "folder"
log_sink_name = element(split("/", var.root_node), 0) == "organizations" ? "logs-audit-org-${local.parent_numeric_id}" : "logs-audit-folder-${local.parent_numeric_id}"
}
/**
* Copyright 2020 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

###############################################################################
# Shared resources folder #
# Terraform top-level resources #
###############################################################################

# Shared folder

module "shared-folder" {
source = "terraform-google-modules/folders/google"
version = "2.0.2"
parent = var.root_node
names = ["shared"]
source = "../../modules/folders"
parent = var.root_node
names = ["shared"]
}

###############################################################################
# Terraform top-level resources #
###############################################################################

# Terraform project

module "project-tf" {
source = "terraform-google-modules/project-factory/google//modules/fabric-project"
version = "5.0.0"
parent = module.shared-folder.id
billing_account = var.billing_account_id
prefix = var.prefix
name = "terraform"
lien_reason = "terraform"
owners = var.terraform_owners
activate_apis = var.project_services
}

# Per environment service accounts

module "service-accounts-tf-environments" {
source = "terraform-google-modules/service-accounts/google"
version = "2.0.2"
project_id = module.project-tf.project_id
org_id = var.organization_id
billing_account_id = var.billing_account_id
prefix = var.prefix
names = var.environments
grant_billing_role = true
generate_keys = var.generate_service_account_keys
module "tf-project" {
source = "../../modules/project"
name = "terraform"
parent = module.shared-folder.id
prefix = var.prefix
billing_account = var.billing_account_id
iam_additive_members = { "roles/owner" = var.iam_terraform_owners }
iam_additive_roles = ["roles/owner"]
services = var.project_services
}

# Bootstrap Terraform state GCS bucket

module "gcs-tf-bootstrap" {
source = "terraform-google-modules/cloud-storage/google"
version = "1.0.0"
project_id = module.project-tf.project_id
prefix = "${var.prefix}-tf"
module "tf-gcs-bootstrap" {
source = "../../modules/gcs"
project_id = module.tf-project.project_id
names = ["tf-bootstrap"]
location = var.gcs_location
}

# Per environment Terraform state GCS buckets

module "gcs-tf-environments" {
source = "terraform-google-modules/cloud-storage/google"
version = "1.0.0"
project_id = module.project-tf.project_id
prefix = "${var.prefix}-tf"
names = var.environments
location = var.gcs_location
set_admin_roles = true
bucket_admins = zipmap(
var.environments,
module.service-accounts-tf-environments.iam_emails_list
)
prefix = "${var.prefix}-tf"
location = var.gcs_defaults.location
}

###############################################################################
# Business units #
# Business units #
###############################################################################

# Business unit 1

module "business-unit-1-folders" {
source = "./modules/business-unit-folders"
business_unit_folder_name = var.business_unit_1_name
environments = var.environments
per_folder_admins = module.service-accounts-tf-environments.iam_emails_list
root_node = var.root_node

}

# Business unit 2

module "business-unit-2-folders" {
source = "./modules/business-unit-folders"
business_unit_folder_name = var.business_unit_2_name
environments = var.environments
per_folder_admins = module.service-accounts-tf-environments.iam_emails_list
root_node = var.root_node

module "bu-business-intelligence" {
source = "../../modules/folders-unit"
name = "Business Intelligence"
short_name = "bi"
automation_project_id = module.tf-project.project_id
billing_account_id = var.billing_account_id
environments = var.environments
gcs_defaults = var.gcs_defaults
organization_id = var.organization_id
root_node = var.root_node
# extra variables from the folders-unit module can be used here to grant
# IAM roles to the bu users, configure the automation service accounts, etc.
# iam_roles = ["viewer"]
# iam_members = { viewer = ["user:[email protected]"] }
}

# Business unit 3

module "business-unit-3-folders" {
source = "./modules/business-unit-folders"
business_unit_folder_name = var.business_unit_3_name
environments = var.environments
per_folder_admins = module.service-accounts-tf-environments.iam_emails_list
root_node = var.root_node

module "bu-machine-learning" {
source = "../../modules/folders-unit"
name = "Machine Learning"
short_name = "ml"
automation_project_id = module.tf-project.project_id
billing_account_id = var.billing_account_id
environments = var.environments
gcs_defaults = var.gcs_defaults
organization_id = var.organization_id
root_node = var.root_node
# extra variables from the folders-unit module can be used here to grant
# IAM roles to the bu users, configure the automation service accounts, etc.
}

###############################################################################
Expand All @@ -133,42 +89,50 @@ module "business-unit-3-folders" {

# Audit logs project

module "project-audit" {
source = "terraform-google-modules/project-factory/google//modules/fabric-project"
version = "5.0.0"
parent = module.shared-folder.id
billing_account = var.billing_account_id
prefix = var.prefix
module "audit-project" {
source = "../../modules/project"
name = "audit"
lien_reason = "audit"
viewers = var.audit_viewers
activate_apis = concat(var.project_services, [
parent = var.root_node
prefix = var.prefix
billing_account = var.billing_account_id
iam_members = {
"roles/bigquery.dataEditor" = [module.audit-log-sinks.writer_identities[0]]
"roles/viewer" = var.iam_audit_viewers
}
iam_roles = [
"roles/bigquery.dataEditor",
"roles/viewer"
]
services = concat(var.project_services, [
"bigquery.googleapis.com",
])
}

# Audit logs destination on BigQuery

module "bq-audit-export" {
source = "terraform-google-modules/log-export/google//modules/bigquery"
version = "3.2.0"
project_id = module.project-audit.project_id
dataset_name = "${replace(local.log_sink_name, "-", "_")}"
log_sink_writer_identity = module.log-sink-audit.writer_identity
# audit logs dataset and sink

module "audit-datasets" {
source = "../../modules/bigquery"
project_id = module.audit-project.project_id
datasets = {
audit_export = {
name = "Audit logs export."
description = "Terraform managed."
location = "EU"
labels = null
options = null
}
}
}

# Audit log sink for root node

module "log-sink-audit" {
source = "terraform-google-modules/log-export/google"
version = "3.2.0"
filter = "logName: \"/logs/cloudaudit.googleapis.com%2Factivity\" OR logName: \"/logs/cloudaudit.googleapis.com%2Fsystem_event\""
log_sink_name = local.log_sink_name
parent_resource_type = local.log_sink_parent_resource_type
parent_resource_id = local.parent_numeric_id
include_children = "true"
unique_writer_identity = "true"
destination_uri = "${module.bq-audit-export.destination_uri}"
module "audit-log-sinks" {
source = "../../modules/logging-sinks"
parent = var.root_node
destinations = {
audit-logs = "bigquery.googleapis.com/projects/${module.audit-project.project_id}/datasets/${module.audit-datasets.names[0]}"
}
sinks = {
audit-logs = var.audit_filter
}
}

###############################################################################
Expand All @@ -177,17 +141,19 @@ module "log-sink-audit" {

# Shared resources project

module "project-shared-resources" {
source = "terraform-google-modules/project-factory/google//modules/fabric-project"
version = "5.0.0"
parent = module.shared-folder.id
billing_account = var.billing_account_id
prefix = var.prefix
name = "shared"
lien_reason = "shared"
activate_apis = var.project_services
extra_bindings_roles = var.shared_bindings_roles
extra_bindings_members = var.shared_bindings_members
module "shared-project" {
source = "../../modules/project"
name = "shared"
parent = module.shared-folder.id
prefix = var.prefix
billing_account = var.billing_account_id
iam_additive_members = {
"roles/owner" = var.iam_shared_owners
}
iam_additive_roles = [
"roles/owner"
]
services = var.project_services
}

# Add further modules here for resources that are common to all business units
Expand Down
Loading

0 comments on commit 708ca97

Please sign in to comment.