Skip to content

Commit

Permalink
Merge new modules list and environments foundation example (#30)
Browse files Browse the repository at this point in the history
* gke-cluster

* net-vpc module and tests

* add TODO to net-vpc module

* add minimal README files with input/output variables to gke and net-vpc modules

* BigQuery Module (#24)

* Bigquery Module

* Added README file

* Added type hints

* gke-cluster

* net-vpc module and tests

* add TODO to net-vpc module

* add minimal README files with input/output variables to gke and net-vpc modules

* BigQuery Module (#24)

* Bigquery Module

* Added README file

* Added type hints

* GCS module

* net vpc module: improve secondary range outputs

* net vpc module: add serve project registration

* project module

* move bigquery module to not-ready folder

* folders module

* rename project module's iam variables

* slight tweak to folder module outputs

* gcs module

* simplify net-vpc module variables

* fix module tests configurations, fix net-vpc module tests

* add pydoc utility

* add/update module READMEs

* add/update module READMEs

* add/update module READMEs

* improve variable type summary generation in tfdoc

* tfdoc: add support for replacing doc in README.md files

* improve module READMEs

* net-vpc-firewall module

* add support for sensitive output attribute in tfdoc

* remove empty function from tfdoc

* render variable type as code in tfdoc

* update module READMEs

* net address module

* net cloudnat module

* remove redundant variable from net-cloudnat module

* vpc module: add support for peering, use network name as subnet name prefix

* net-vpn-static module

* net-vpn-static module README

* net-vpn-static module README

* tfdoc: fix error on undeclared variable type

* dns module

* set version for all modules

* kms module (untested)

* change kms key self links output to map, fix gcs and kms iam variable descriptions

* fix kms module

* update kms module readme

* simplify local iam pairs in modules

* service accounts module (unfinished)

* work on service accounts module

* project module: add gcr service account

* project module: update outputs in README

* first working version of the iam service accounts module

* iam service accounts module: extra checks in locals

* modules/net-cloudnat: reorder variables

* modules/net-vpn-dynamic: initial import (untested)

* modules/net-vpn-dynamic: first working version

* modules/net-vpn-dynamic: add outputs for auto-created router

* modules/net-vpn-dynamic: update README

* modules/net-[vpn,cloudnat]: clean up variable,s remove prefix

* modules/net-vpn-dynamic: add advertisement configuration to tunnel bgp peer, refactor variables

* tfdoc: add tooltips for variable types and defaults

* modules: update README variables and outputs

* tfdoc: improve variable default rendering

* modules: update README variables and outputs

* modules/net-vpc: minimal output refactoring

* modules/vm-cos: initial import, base resources working, no outputs

* modules/vm-cos: add variable descriptions

* tfdoc: fix parsing in type and default blocks

* modules/vm-cos: fix README

* tfdoc: fix parsing in type and default blocks

* modules/vm-cos: fix README

* modules/compute-vm: initial working import (not fully tested)

* modules/vm-cos: move to not-ready

* tfdoc: fix variable defaults formatting

* modules: update README files with tfdoc fixes

* modules: add initial examples

* gke-nodepool: initial import, untested

* gke nodepool: add README, fix location variable, set node count default to 1

* gke cluster: fix private cluster variables

* gke nodepool: fix README title

* gke cluster: add output for cluster location

* gke nodepool: add missing variables for project id and cluster name, remove default from location variable, fix gke version assignment

* gke nodepool: update README

* net-cloudnat: fix router name when creating default router

* fix variables used for address and router optional creation

* vpn dynamic: fix README

* modules/net-vpn-dynamic: fix router name output

* modules/compute-vm: remove unused variable

* modules/compute-vm-cos-coredns: initial import

* Update foundations modules versions (#26)

* update foundations modules versions

* update Terraform version to v0.12.19 in CI test configuration

* backport tfdoc from Ludo's branch (#27)

* Update docs using tfdoc format (#28)

* update README files

* set all types on variables

* foundations/environments: move log filter to a variable, use org for xpn by default

* foundations/environments: do not use liens by default

* modules/ntp-vpc: better shared_vpc_host variable description

* modules/logging-sinks: initial version

* modules/logging-sinks: streamline options in sinks variable

* modules/compute-vm-cos-coredns: add support for additional files

* modules/folders: rename from 'folder'

* modules/logging-sinks: fix circular dependencies and improve variables

* modules/project: remove extra variable

* modules/bigquery: new module with dataset support only

* foundations/environments: refactor using local modules

* modules/bigquery: better variables, README description and example

* modules: fix a few READMEs

Co-authored-by: Julio Castillo <[email protected]>
  • Loading branch information
ludoo and juliocc authored Jan 17, 2020
1 parent 2e07d17 commit 8496243
Show file tree
Hide file tree
Showing 127 changed files with 8,157 additions and 262 deletions.
2 changes: 1 addition & 1 deletion .ci/cloudbuild.test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ steps:
- PYTHONDONTWRITEBYTECODE=true

substitutions:
_TERRAFORM_VERSION: 0.12.8
_TERRAFORM_VERSION: 0.12.19

tags:
- "ci"
Expand Down
75 changes: 37 additions & 38 deletions foundations/business-units/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,45 +25,44 @@ The number of resources in this sample is kept to a minimum so as to make it gen

This sample uses a top-level folder to encapsulate projects that host resources that are not specific to a single environment. If no shared services are needed,the Terraform and audit modules can be easily attached to the root node, and the shared services folder and project removed from `main.tf`.

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Inputs
<!-- BEGIN TFDOC -->
## Variables

| Name | Description | Type | Default | Required |
|------|-------------|:----:|:-----:|:-----:|
| audit\_viewers | Audit project viewers, in IAM format. | list | `<list>` | no |
| billing\_account\_id | Billing account id used as default for new projects. | string | n/a | yes |
| business\_unit\_1\_name | Business unit 1 short name. | string | n/a | yes |
| business\_unit\_2\_name | Business unit 2 short name. | string | n/a | yes |
| business\_unit\_3\_name | Business unit 3 short name. | string | n/a | yes |
| environments | Environment short names. | list(string) | n/a | yes |
| gcs\_location | GCS bucket location. | string | `"EU"` | no |
| generate\_service\_account\_keys | Generate and store service account keys in the state file. | string | `"false"` | no |
| organization\_id | Organization id. | string | n/a | yes |
| prefix | Prefix used for resources that need unique names. | string | n/a | yes |
| project\_services | Service APIs enabled by default in new projects. | list | `<list>` | no |
| root\_node | Root node for the new hierarchy, either 'organizations/org_id' or 'folders/folder_id'. | string | n/a | yes |
| shared\_bindings\_members | List of comma-delimited IAM-format members for the additional shared project bindings. | list | `<list>` | no |
| shared\_bindings\_roles | List of roles for additional shared project bindings. | list | `<list>` | no |
| terraform\_owners | Terraform project owners, in IAM format. | list | `<list>` | no |
| name | description | type | required | default |
|---|---|:---: |:---:|:---:|
| billing_account_id | Billing account id used as default for new projects. | <code title="">string</code> | | |
| business_unit_1_name | Business unit 1 short name. | <code title="">string</code> | | |
| business_unit_2_name | Business unit 2 short name. | <code title="">string</code> | | |
| business_unit_3_name | Business unit 3 short name. | <code title="">string</code> | | |
| environments | Environment short names. | <code title="list&#40;string&#41;">list(string)</code> | | |
| organization_id | Organization id. | <code title="">string</code> | | |
| prefix | Prefix used for resources that need unique names. | <code title="">string</code> | | |
| root_node | Root node for the new hierarchy, either 'organizations/org_id' or 'folders/folder_id'. | <code title="">string</code> | | |
| *audit_viewers* | Audit project viewers, in IAM format. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |
| *gcs_location* | GCS bucket location. | <code title="">string</code> | | <code title="">EU</code> |
| *generate_service_account_keys* | Generate and store service account keys in the state file. | <code title="">bool</code> | | <code title="">false</code> |
| *project_services* | Service APIs enabled by default in new projects. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="&#91;&#10;&#34;resourceviews.googleapis.com&#34;,&#10;&#34;stackdriver.googleapis.com&#34;,&#10;&#93;">...</code> |
| *shared_bindings_members* | List of comma-delimited IAM-format members for the additional shared project bindings. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |
| *shared_bindings_roles* | List of roles for additional shared project bindings. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |
| *terraform_owners* | Terraform project owners, in IAM format. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |

## Outputs

| Name | Description |
|------|-------------|
| audit\_logs\_bq\_dataset | Bigquery dataset for the audit logs export. |
| audit\_logs\_project | Project that holds the audit logs export resources. |
| bootstrap\_tf\_gcs\_bucket | GCS bucket used for the bootstrap Terraform state. |
| business\_unit\_1\_environment\_folders\_ids | Business unit 1 environment folders. |
| business\_unit\_1\_folder\_id | Business unit 1 top-level folder ID. |
| business\_unit\_2\_environment\_folders\_ids | Business unit 2 environment folders. |
| business\_unit\_2\_folder\_id | Business unit 2 top-level folder ID. |
| business\_unit\_3\_environment\_folders\_ids | Business unit 3 environment folders. |
| business\_unit\_3\_folder\_id | Business unit 3 top-level folder ID. |
| environment\_service\_account\_keys | Service account keys used to run each environment Terraform modules. |
| environment\_service\_accounts | Service accounts used to run each environment Terraform modules. |
| environment\_tf\_gcs\_buckets | GCS buckets used for each environment Terraform state. |
| shared\_folder\_id | Shared folder ID. |
| shared\_resources\_project | Project that holdes resources shared across business units. |
| terraform\_project | Project that holds the base Terraform resources. |

<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
| name | description | sensitive |
|---|---|:---:|
| audit_logs_bq_dataset | Bigquery dataset for the audit logs export. | |
| audit_logs_project | Project that holds the audit logs export resources. | |
| bootstrap_tf_gcs_bucket | GCS bucket used for the bootstrap Terraform state. | |
| business_unit_1_environment_folders_ids | Business unit 1 environment folders. | |
| business_unit_1_folder_id | Business unit 1 top-level folder ID. | |
| business_unit_2_environment_folders_ids | Business unit 2 environment folders. | |
| business_unit_2_folder_id | Business unit 2 top-level folder ID. | |
| business_unit_3_environment_folders_ids | Business unit 3 environment folders. | |
| business_unit_3_folder_id | Business unit 3 top-level folder ID. | |
| environment_service_account_keys | Service account keys used to run each environment Terraform modules. ||
| environment_service_accounts | Service accounts used to run each environment Terraform modules. | |
| environment_tf_gcs_buckets | GCS buckets used for each environment Terraform state. | |
| shared_folder_id | Shared folder ID. | |
| shared_resources_project | Project that holdes resources shared across business units. | |
| terraform_project | Project that holds the base Terraform resources. | |
<!-- END TFDOC -->
8 changes: 4 additions & 4 deletions foundations/business-units/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ locals {

module "shared-folder" {
source = "terraform-google-modules/folders/google"
version = "2.0.0"
version = "2.0.2"
parent = var.root_node
names = ["shared"]
}
Expand All @@ -53,7 +53,7 @@ module "project-tf" {

module "service-accounts-tf-environments" {
source = "terraform-google-modules/service-accounts/google"
version = "2.0.1"
version = "2.0.2"
project_id = module.project-tf.project_id
org_id = var.organization_id
billing_account_id = var.billing_account_id
Expand Down Expand Up @@ -151,7 +151,7 @@ module "project-audit" {

module "bq-audit-export" {
source = "terraform-google-modules/log-export/google//modules/bigquery"
version = "3.1.0"
version = "3.2.0"
project_id = module.project-audit.project_id
dataset_name = "${replace(local.log_sink_name, "-", "_")}"
log_sink_writer_identity = module.log-sink-audit.writer_identity
Expand All @@ -161,7 +161,7 @@ module "bq-audit-export" {

module "log-sink-audit" {
source = "terraform-google-modules/log-export/google"
version = "3.1.0"
version = "3.2.0"
filter = "logName: \"/logs/cloudaudit.googleapis.com%2Factivity\" OR logName: \"/logs/cloudaudit.googleapis.com%2Fsystem_event\""
log_sink_name = local.log_sink_name
parent_resource_type = local.log_sink_parent_resource_type
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@

module "business-unit-folder" {
source = "terraform-google-modules/folders/google"
version = "2.0.0"
version = "2.0.2"
parent = var.root_node
names = [var.business_unit_folder_name]
}
Expand All @@ -29,7 +29,7 @@ module "business-unit-folder" {

module "environment-folders" {
source = "terraform-google-modules/folders/google"
version = "2.0.0"
version = "2.0.2"
parent = module.business-unit-folder.id
names = var.environments
set_roles = true
Expand All @@ -41,4 +41,4 @@ module "environment-folders" {
"roles/compute.networkAdmin",
"roles/compute.xpnAdmin"
]
}
}
7 changes: 7 additions & 0 deletions foundations/business-units/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@

variable "audit_viewers" {
description = "Audit project viewers, in IAM format."
type = list(string)
default = []
}

Expand Down Expand Up @@ -44,11 +45,13 @@ variable "environments" {

variable "generate_service_account_keys" {
description = "Generate and store service account keys in the state file."
type = bool
default = false
}

variable "gcs_location" {
description = "GCS bucket location."
type = string
default = "EU"
}

Expand All @@ -70,21 +73,25 @@ variable "root_node" {
variable "shared_bindings_members" {
description = "List of comma-delimited IAM-format members for the additional shared project bindings."
# example: ["user:[email protected],[email protected]", "user:[email protected]"]
type = list(string)
default = []
}
variable "shared_bindings_roles" {
description = "List of roles for additional shared project bindings."
# example: ["roles/storage.objectViewer", "roles/storage.admin"]
type = list(string)
default = []
}

variable "terraform_owners" {
description = "Terraform project owners, in IAM format."
type = list(string)
default = []
}

variable "project_services" {
description = "Service APIs enabled by default in new projects."
type = list(string)
default = [
"resourceviews.googleapis.com",
"stackdriver.googleapis.com",
Expand Down
63 changes: 31 additions & 32 deletions foundations/environments/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,38 +27,37 @@ For more complex setups where multiple shared services projects are needed to en

If no shared services are needed, the shared service project module can of course be removed from `main.tf`.

<!-- BEGINNING OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
## Inputs

| Name | Description | Type | Default | Required |
|------|-------------|:----:|:-----:|:-----:|
| audit\_viewers | Audit project viewers, in IAM format. | list | `<list>` | no |
| billing\_account\_id | Billing account id used as default for new projects. | string | n/a | yes |
| environments | Environment short names. | list(string) | n/a | yes |
| gcs\_location | GCS bucket location. | string | `"EU"` | no |
| generate\_service\_account\_keys | Generate and store service account keys in the state file. | string | `"false"` | no |
| grant\_xpn\_folder\_roles | Grant roles needed for Shared VPC creation to service accounts at the environment folder level. | string | `"true"` | no |
| grant\_xpn\_org\_roles | Grant roles needed for Shared VPC creation to service accounts at the organization level. | string | `"false"` | no |
| organization\_id | Organization id. | string | n/a | yes |
| prefix | Prefix used for resources that need unique names. | string | n/a | yes |
| project\_services | Service APIs enabled by default in new projects. | list | `<list>` | no |
| root\_node | Root node for the new hierarchy, either 'organizations/org_id' or 'folders/folder_id'. | string | n/a | yes |
| shared\_bindings\_members | List of comma-delimited IAM-format members for the additional shared project bindings. | list | `<list>` | no |
| shared\_bindings\_roles | List of roles for additional shared project bindings. | list | `<list>` | no |
| terraform\_owners | Terraform project owners, in IAM format. | list | `<list>` | no |
<!-- BEGIN TFDOC -->
## Variables

| name | description | type | required | default |
|---|---|:---: |:---:|:---:|
| billing_account_id | Billing account id used as default for new projects. | <code title="">string</code> | | |
| environments | Environment short names. | <code title="list&#40;string&#41;">list(string)</code> | | |
| organization_id | Organization id. | <code title="">string</code> | | |
| prefix | Prefix used for resources that need unique names. | <code title="">string</code> | | |
| root_node | Root node for the new hierarchy, either 'organizations/org_id' or 'folders/folder_id'. | <code title="">string</code> | | |
| *audit_viewers* | Audit project viewers, in IAM format. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |
| *gcs_location* | GCS bucket location. | <code title="">string</code> | | <code title="">EU</code> |
| *generate_service_account_keys* | Generate and store service account keys in the state file. | <code title="">bool</code> | | <code title="">false</code> |
| *grant_xpn_folder_roles* | Grant roles needed for Shared VPC creation to service accounts at the environment folder level. | <code title="">bool</code> | | <code title="">true</code> |
| *grant_xpn_org_roles* | Grant roles needed for Shared VPC creation to service accounts at the organization level. | <code title="">bool</code> | | <code title="">false</code> |
| *project_services* | Service APIs enabled by default in new projects. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="&#91;&#10;&#34;resourceviews.googleapis.com&#34;,&#10;&#34;stackdriver.googleapis.com&#34;,&#10;&#93;">...</code> |
| *shared_bindings_members* | List of comma-delimited IAM-format members for the additional shared project bindings. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |
| *shared_bindings_roles* | List of roles for additional shared project bindings. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |
| *terraform_owners* | Terraform project owners, in IAM format. | <code title="list&#40;string&#41;">list(string)</code> | | <code title="">[]</code> |

## Outputs

| Name | Description |
|------|-------------|
| audit\_logs\_bq\_dataset | Bigquery dataset for the audit logs export. |
| audit\_logs\_project | Project that holds the audit logs export resources. |
| bootstrap\_tf\_gcs\_bucket | GCS bucket used for the bootstrap Terraform state. |
| environment\_folders | Top-level environment folders. |
| environment\_service\_account\_keys | Service account keys used to run each environment Terraform modules. |
| environment\_service\_accounts | Service accounts used to run each environment Terraform modules. |
| environment\_tf\_gcs\_buckets | GCS buckets used for each environment Terraform state. |
| shared\_resources\_project | Project that holdes resources shared across environments. |
| terraform\_project | Project that holds the base Terraform resources. |

<!-- END OF PRE-COMMIT-TERRAFORM DOCS HOOK -->
| name | description | sensitive |
|---|---|:---:|
| audit_logs_bq_dataset | Bigquery dataset for the audit logs export. | |
| audit_logs_project | Project that holds the audit logs export resources. | |
| bootstrap_tf_gcs_bucket | GCS bucket used for the bootstrap Terraform state. | |
| environment_folders | Top-level environment folders. | |
| environment_service_account_keys | Service account keys used to run each environment Terraform modules. ||
| environment_service_accounts | Service accounts used to run each environment Terraform modules. | |
| environment_tf_gcs_buckets | GCS buckets used for each environment Terraform state. | |
| shared_resources_project | Project that holdes resources shared across environments. | |
| terraform_project | Project that holds the base Terraform resources. | |
<!-- END TFDOC -->
38 changes: 38 additions & 0 deletions foundations/environments/locals.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
/**
* Copyright 2019 Google LLC
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

locals {
folder_roles = concat(var.iam_folder_roles, local.sa_xpn_folder_role)
sa_billing_account_role = (
var.iam_billing_config.target_org ? [] : ["roles/billing.user"]
)
sa_billing_org_role = (
! var.iam_billing_config.target_org ? [] : ["roles/billing.user"]
)
sa_xpn_folder_role = (
local.sa_xpn_target_org ? [] : ["roles/compute.xpnAdmin"]
)
sa_xpn_org_roles = (
local.sa_xpn_target_org
? ["roles/compute.xpnAdmin", "roles/resourcemanager.organizationViewer"]
: ["roles/resourcemanager.organizationViewer"]
)
sa_xpn_target_org = (
var.iam_xpn_config.target_org
||
substr(var.root_node, 0, 13) == "organizations"
)
}
Loading

0 comments on commit 8496243

Please sign in to comment.