From 5a59d38ee9aa3b35ac685308b99883ba8ba18458 Mon Sep 17 00:00:00 2001 From: Ludovico Magnocavallo Date: Fri, 3 Apr 2020 14:06:48 +0200 Subject: [PATCH] Merge development branch (#44) * VPN-HA module initial commit * Added readme for net-vpn-ha module * Update readme, add simple description * Merge new modules list and environments foundation example (#30) * gke-cluster * net-vpc module and tests * add TODO to net-vpc module * add minimal README files with input/output variables to gke and net-vpc modules * BigQuery Module (#24) * Bigquery Module * Added README file * Added type hints * gke-cluster * net-vpc module and tests * add TODO to net-vpc module * add minimal README files with input/output variables to gke and net-vpc modules * BigQuery Module (#24) * Bigquery Module * Added README file * Added type hints * GCS module * net vpc module: improve secondary range outputs * net vpc module: add serve project registration * project module * move bigquery module to not-ready folder * folders module * rename project module's iam variables * slight tweak to folder module outputs * gcs module * simplify net-vpc module variables * fix module tests configurations, fix net-vpc module tests * add pydoc utility * add/update module READMEs * add/update module READMEs * add/update module READMEs * improve variable type summary generation in tfdoc * tfdoc: add support for replacing doc in README.md files * improve module READMEs * net-vpc-firewall module * add support for sensitive output attribute in tfdoc * remove empty function from tfdoc * render variable type as code in tfdoc * update module READMEs * net address module * net cloudnat module * remove redundant variable from net-cloudnat module * vpc module: add support for peering, use network name as subnet name prefix * net-vpn-static module * net-vpn-static module README * net-vpn-static module README * tfdoc: fix error on undeclared variable type * dns module * set version for all modules * kms module (untested) * change kms key self links output to map, fix gcs and kms iam variable descriptions * fix kms module * update kms module readme * simplify local iam pairs in modules * service accounts module (unfinished) * work on service accounts module * project module: add gcr service account * project module: update outputs in README * first working version of the iam service accounts module * iam service accounts module: extra checks in locals * modules/net-cloudnat: reorder variables * modules/net-vpn-dynamic: initial import (untested) * modules/net-vpn-dynamic: first working version * modules/net-vpn-dynamic: add outputs for auto-created router * modules/net-vpn-dynamic: update README * modules/net-[vpn,cloudnat]: clean up variable,s remove prefix * modules/net-vpn-dynamic: add advertisement configuration to tunnel bgp peer, refactor variables * tfdoc: add tooltips for variable types and defaults * modules: update README variables and outputs * tfdoc: improve variable default rendering * modules: update README variables and outputs * modules/net-vpc: minimal output refactoring * modules/vm-cos: initial import, base resources working, no outputs * modules/vm-cos: add variable descriptions * tfdoc: fix parsing in type and default blocks * modules/vm-cos: fix README * tfdoc: fix parsing in type and default blocks * modules/vm-cos: fix README * modules/compute-vm: initial working import (not fully tested) * modules/vm-cos: move to not-ready * tfdoc: fix variable defaults formatting * modules: update README files with tfdoc fixes * modules: add initial examples * gke-nodepool: initial import, untested * gke nodepool: add README, fix location variable, set node count default to 1 * gke cluster: fix private cluster variables * gke nodepool: fix README title * gke cluster: add output for cluster location * gke nodepool: add missing variables for project id and cluster name, remove default from location variable, fix gke version assignment * gke nodepool: update README * net-cloudnat: fix router name when creating default router * fix variables used for address and router optional creation * vpn dynamic: fix README * modules/net-vpn-dynamic: fix router name output * modules/compute-vm: remove unused variable * modules/compute-vm-cos-coredns: initial import * Update foundations modules versions (#26) * update foundations modules versions * update Terraform version to v0.12.19 in CI test configuration * backport tfdoc from Ludo's branch (#27) * Update docs using tfdoc format (#28) * update README files * set all types on variables * foundations/environments: move log filter to a variable, use org for xpn by default * foundations/environments: do not use liens by default * modules/ntp-vpc: better shared_vpc_host variable description * modules/logging-sinks: initial version * modules/logging-sinks: streamline options in sinks variable * modules/compute-vm-cos-coredns: add support for additional files * modules/folders: rename from 'folder' * modules/logging-sinks: fix circular dependencies and improve variables * modules/project: remove extra variable * modules/bigquery: new module with dataset support only * foundations/environments: refactor using local modules * modules/bigquery: better variables, README description and example * modules: fix a few READMEs Co-authored-by: Julio Castillo * modules/net-vpc: README description and examples * modules/net-vpc: tweak README description and examples * modules/net-vpc: tweak README description and examples * modules/net-vpc-firewall: change tag-based rule default ranges, improve README examples and description * modules/compute-vm: README changes * modules/compute-vm: use an object for the service account variable, update README * modules/compute-vm: update README variables table * modules/compute-vm: add TODO list to README * modules/compute-vm: add TODO list to README * modules/compute-vm: add outputs for service account * modules/net-cloudnat: README * modules/net-cloudnat: README * modules/net-cloudnat: add router_create variable * modules/compute-vm: simplify service account variables * modules/net-vpn-dynamic: fix README example, use local secret for both empty string and null * modules/net-vpn-dynamic: improve README example * modules/gke-cluster: minimal README tweaks * modules/kms: fix ephemeral keys resource name * modules/iam-service-accounts: add storage roles * modules/gke-nodepool: fix node default scopes * New project variable to prevent deletion of default network (#32) * New project variable to prevent deletion of default network This is a workaround to fix terraform-google-modules/cloud-foundation-fabric#31 while the GCP terraform provider is fixed * Add TODOs to remove workarounds in the project module * Fix Cloud Build files * modules/gke-nodepool: add monitoring scope to defaults * modules/iam-service-accounts: add support for IAM bindings onthe service accounts * playground module in sandbox, remove not ready modules * Fix ci configurations in development branch (#33) * try fixing ci confgurations * add exclusion match to ci boilerplate check * add skip boilerplate comment to compute-vm-cos-coredns template fragment * modules/gke-cluster: fix boilerplate in outputs * Simplify tests, re-enable CI * add instance group support to compute-vm, start tests refactoring * modules/compute-vm: group fixes, tests * modules/compute-vm: minimal test beautification * simplify top-level pytest fixture * modules/dns: tests and minor tweaks * fix missing boilerplate in tests * re-add requirements file to tests folder * re-enable tests in ci build configuration * Folder module tests and fixes (#38) * folder tests wip * modules/folders: tests and tweaks * update folders and compute-vm README files * modules/gcs: tests and minor tweaks * Create README.md * Update README.md * Update README.md * Update README.md * Added docker image for strongSwan * Add support for routes and tests to net-vpc module (#39) * modules/net-vpc: add routes (untested) * initial tests * modules/net-vpc: add test for flow logs * modules/net-vpc: split tests into two separate files * modules/net-vpc: routes test * modules/net-vpc: test routes * Add support for Terraform plugin cache in ci test build file (#40) * add Terraform plugin caching to test ci build configuration * fix mkdir in test build configuration * trigger test check * Refactor dynamic vpn configuration for on-prem-in-a-box module * Fix dynamic vpn for onprem-in-a-box module * Migrate Shared VPC example to local modules (#41) * wip * wip * validated, untested * modules/compute-vm: make service account email in locals resilient to destroy * modules/project: make project id output depend on iam roles * fixes * shared-vpc tweaks * update diagram * update README input output tables * modules/compute-vm: add service account IAM email output * move GKE service account roles at the project level, add GCE service account roles * update diagram and README * modules/project: add extra output for IAM-dependent project id * update modules READMEs * minor tweaks * modules/compute-vm: fix service account output * remove static address from NAT * fix container service agent binding dependency * rename shared vpc * Update README.md * Update README.md * Add static vpn gw to on-prem-in-a-box module * Refactor hub and spoke to use new modules (#42) * modules/compute-vm: saner defaults for service account scopes * hub and spoke refactor, docs still missing * complete hub and spoke * Update README.md * Add toolbox docker container, fix gw routing to the internet * Add DNS Hybrid connectivity parameters * Fix onprem dns zone for the static vpn configuration * Added readme.md for on-prem module * Add new line at the end of the files * Add boilerplate for cloudbuild config files * fix boilerplate in strongswan shell script * Update README.md * include missing file to fix merge conflict * remove missing file to fix merge conflict * include missing file to fix merge conflict (again) * remove content from spurious file used to avoid merge conflicts * Add net-vpc-peering module * Initial commit for hub-and-spoke-peering infrastructure example * Fix typos in infrastructure/ READMEs * remove stale file * use larger resolution version of hub and spoke diagram * Update README.md * Update hub-and-spoke-peerings example to use internal modules * Add initial project tests (#46) * modules/project: make prefix optional * initial project module tests * modules/project: use null for unset parent * modules/dns: backport PR6 from the CFT dns module * Add testing resources including on-prem-in-a-box to hub-and-spoke-peerings example * Fix firewall rules to allow connectivity, switch to custom route advertisement for onprem -> spokes connectivity * Move locals out of main.tf * remove ssh tag from compute-vm variable default * Add ssh tag to the test vms * Update README.md * Update README.md * Update README.md * Hub and spoke peering changes (#48) * rename hub-and-spoke-vpn * add ssh tag to shared-vpc-gke instance * rename and rework hub and spoke peering * fix test requirements * align hub and spoke peering with module contents * diagram * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * minimal fixes to onprem examples variable files * onprem example stub, missing DNS zones and private.googleapis records onprem * add missing boilerplate * Update README.md * Update README.md * infra/onprem: add test instance and minimal outputs * add DNS modules and resource * infra/onprem: diagram and initial README * minor changes to onprem module and example (#49) * update toolbox image * infra/onprem: add zone for private access, add metadata domain to onprem dns * infra/onprem: onnprem service account, add testing procedure in README * Update README.md * infra/onprem: remove extra variable * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * infra/onprem: rename forwarder address variable * Update README: Added explicit --tunnel-through-iap for gcloud compute ssh commands * Update top-level and section READMEs (#50) * top-level README WIP * rewrite top-level README * change top-level README title * remove initial quote in top-level README * Update README.md * Update README.md * Update README.md * foundations README * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * add experimental scheduled cloud function module * scheduled cloud function module: allow disabling schedule * business-units foundation example (#52) * Added folder-units module. * Business units example update (WIP) * Update all BU modules to internal ones * Refactoring business-units example, add billing and org IAM handling * update projects tests for new iam additive naming * update project README for new iam additive naming * streamline bu example and module (#53) Co-authored-by: Ludovico Magnocavallo * align net-vpn-ha interface with the other vpn modules * update module README files * Update README.md * Update README.md * Create CHANGELOG.md * Refactor COS module to be generic (#51) * Create generic COS module and update CoreDNS module to use it * Update compute-vm-cos README * Fix COS README * Update COS example * Skip boilerplate check for COS file template * Make COS module more generic and provide preset configurations * Update COS module documentation * tfdoc: add support for multiple variables files * compute-vm: split boot disk in separate variable file for cos module support * Streamline cos modules (#54) * tfdoc: fix bug in last commit * compute-vm: add support for user-data * compute-vm: restore noncos variable split * remove compute-vm-cos-coredns * compute-vm: revert to original state * cos-container/coredns * fix variables mess * cos/coredns fixes * cos/mysql * remove stale compute-vm-cos module * add test instance to cos modules * tfdoc: add support for multiple output files * cos: add initial READMEs * Update README.md * Update README.md * Update README.md * Update README.md * Update README.md * add test apply fixture * cos-coredns: tested * Update README.md * Fix typo * cos-coredns: refactor README * Update README.md * test yaml validity in cos modules tests * cos mysql tests * cos mysql: refactor and test (disk tests missing) * onprem: fix Coredns * cos mysql: additional disk working * cos modules: fix instance disks for no instance * update some modules READMEs * update some modules READMEs * Update README.md * Update README.md * add simple tests for foundations/environments * change default for org id in foundations/environments to avoid errors when none is specified * fix null/empty organization id in foundations/environments * fix errors when destroying on empty state in foundations/environments * fundations/bu: fix errors when destroying with empty state * modules/gcs: make outputs resilient on destroy with empty state * modules/folders: make outputs resilient on destroy with empty state * switch organization_id variable to long form in foundations/bu and modules/folders-unit * Update README.md * infra/shared-vpc: remove duplicate tag attribute from bastion Co-authored-by: Aleksandr Averbukh Co-authored-by: Julio Castillo Co-authored-by: Julio Castillo --- README.md | 54 +++++++++++++++++++++++++++++++++++ main.tf | 62 +++++++++++++++++++++++++++++++++++++++++ outputs.tf | 63 +++++++++++++++++++++++++++++++++++++++++ variables.tf | 79 ++++++++++++++++++++++++++++++++++++++++++++++++++++ versions.tf | 19 +++++++++++++ 5 files changed, 277 insertions(+) create mode 100644 README.md create mode 100644 main.tf create mode 100644 outputs.tf create mode 100644 variables.tf create mode 100644 versions.tf diff --git a/README.md b/README.md new file mode 100644 index 0000000..2058365 --- /dev/null +++ b/README.md @@ -0,0 +1,54 @@ +# Google Cloud Storage Module + +## Example + +```hcl +module "buckets" { + source = "./modules/gcs" + project_id = "myproject" + prefix = "test" + names = ["bucket-one", "bucket-two"] + bucket_policy_only = { + bucket-one = false + } + iam_members = { + bucket-two = { + "roles/storage.admin" = ["group:storage@example.com"] + } + } + iam_roles = { + bucket-two = ["roles/storage.admin"] + } +} +``` + + +## Variables + +| name | description | type | required | default | +|---|---|:---: |:---:|:---:| +| names | Bucket name suffixes. | list(string) | ✓ | | +| project_id | Bucket project id. | string | ✓ | | +| *bucket_policy_only* | Optional map to disable object ACLS keyed by name, defaults to true. | map(bool) | | {} | +| *force_destroy* | Optional map to set force destroy keyed by name, defaults to false. | map(bool) | | {} | +| *iam_members* | IAM members keyed by bucket name and role. | map(map(list(string))) | | null | +| *iam_roles* | IAM roles keyed by bucket name. | map(list(string)) | | null | +| *labels* | Labels to be attached to all buckets. | map(string) | | {} | +| *location* | Bucket location. | string | | EU | +| *prefix* | Prefix used to generate the bucket name. | string | | | +| *storage_class* | Bucket storage class. | string | | MULTI_REGIONAL | +| *versioning* | Optional map to set versioning keyed by name, defaults to false. | map(bool) | | {} | + +## Outputs + +| name | description | sensitive | +|---|---|:---:| +| bucket | Bucket resource (for single use). | | +| buckets | Bucket resources. | | +| name | Bucket name (for single use). | | +| names | Bucket names. | | +| names_list | List of bucket names. | | +| url | Bucket URL (for single use). | | +| urls | Bucket URLs. | | +| urls_list | List of bucket URLs. | | + diff --git a/main.tf b/main.tf new file mode 100644 index 0000000..baff11d --- /dev/null +++ b/main.tf @@ -0,0 +1,62 @@ +/** + * Copyright 2019 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + buckets = ( + local.has_buckets + ? [for name in var.names : google_storage_bucket.buckets[name]] + : [] + ) + # needed when destroying + has_buckets = length(google_storage_bucket.buckets) > 0 + iam_pairs = var.iam_roles == null ? [] : flatten([ + for name, roles in var.iam_roles : + [for role in roles : { name = name, role = role }] + ]) + iam_keypairs = { + for pair in local.iam_pairs : + "${pair.name}-${pair.role}" => pair + } + iam_members = var.iam_members == null ? {} : var.iam_members + prefix = var.prefix == "" ? "" : join("-", [var.prefix, lower(var.location), ""]) +} + +resource "google_storage_bucket" "buckets" { + for_each = toset(var.names) + name = "${local.prefix}${lower(each.key)}" + project = var.project_id + location = var.location + storage_class = var.storage_class + force_destroy = lookup(var.force_destroy, each.key, false) + bucket_policy_only = lookup(var.bucket_policy_only, each.key, true) + versioning { + enabled = lookup(var.versioning, each.key, false) + } + labels = merge(var.labels, { + location = lower(var.location) + name = lower(each.key) + storage_class = lower(var.storage_class) + }) +} + +resource "google_storage_bucket_iam_binding" "bindings" { + for_each = local.iam_keypairs + bucket = google_storage_bucket.buckets[each.value.name].name + role = each.value.role + members = lookup( + lookup(local.iam_members, each.value.name, {}), each.value.role, [] + ) +} diff --git a/outputs.tf b/outputs.tf new file mode 100644 index 0000000..9a8f8df --- /dev/null +++ b/outputs.tf @@ -0,0 +1,63 @@ +/** + * Copyright 2018 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +output "bucket" { + description = "Bucket resource (for single use)." + value = local.has_buckets ? local.buckets[0] : null +} + +output "name" { + description = "Bucket name (for single use)." + value = local.has_buckets ? local.buckets[0].name : null +} + +output "url" { + description = "Bucket URL (for single use)." + value = local.has_buckets ? local.buckets[0].url : null +} + +output "buckets" { + description = "Bucket resources." + value = local.buckets +} + +output "names" { + description = "Bucket names." + value = ( + local.has_buckets + ? zipmap(var.names, [for b in local.buckets : lookup(b, "name", null)]) + : {} + ) +} + +output "urls" { + description = "Bucket URLs." + value = ( + local.has_buckets + ? zipmap(var.names, [for b in local.buckets : b.url]) + : {} + ) +} + +output "names_list" { + description = "List of bucket names." + value = [for b in local.buckets : b.name] +} + +output "urls_list" { + description = "List of bucket URLs." + value = [for b in local.buckets : b.name] +} diff --git a/variables.tf b/variables.tf new file mode 100644 index 0000000..6cc712e --- /dev/null +++ b/variables.tf @@ -0,0 +1,79 @@ +/** + * Copyright 2018 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +variable "bucket_policy_only" { + description = "Optional map to disable object ACLS keyed by name, defaults to true." + type = map(bool) + default = {} +} + +variable "force_destroy" { + description = "Optional map to set force destroy keyed by name, defaults to false." + type = map(bool) + default = {} +} + +variable "iam_members" { + description = "IAM members keyed by bucket name and role." + type = map(map(list(string))) + default = null +} + +variable "iam_roles" { + description = "IAM roles keyed by bucket name." + type = map(list(string)) + default = null +} + +variable "labels" { + description = "Labels to be attached to all buckets." + type = map(string) + default = {} +} + +variable "location" { + description = "Bucket location." + type = string + default = "EU" +} + +variable "names" { + description = "Bucket name suffixes." + type = list(string) +} + +variable "prefix" { + description = "Prefix used to generate the bucket name." + type = string + default = "" +} + +variable "project_id" { + description = "Bucket project id." + type = string +} + +variable "storage_class" { + description = "Bucket storage class." + type = string + default = "MULTI_REGIONAL" +} + +variable "versioning" { + description = "Optional map to set versioning keyed by name, defaults to false." + type = map(bool) + default = {} +} diff --git a/versions.tf b/versions.tf new file mode 100644 index 0000000..ce6918e --- /dev/null +++ b/versions.tf @@ -0,0 +1,19 @@ +/** + * Copyright 2019 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +terraform { + required_version = ">= 0.12.6" +}