diff --git a/1-bootstrap/README.md b/1-bootstrap/README.md index 2326db2b..2f95fa75 100644 --- a/1-bootstrap/README.md +++ b/1-bootstrap/README.md @@ -1,15 +1,16 @@ # 1. Bootstrap phase - The bootstrap phase establishes the 3 initial pipelines of the Enterprise Application blueprint. These pipelines are: + - the Multitenant Infrastructure pipeline - the Application Factory - the Fleet-Scope pipeline An overview of the deployment methodology for the Enterprise Application blueprint is shown below. -![Enterprise Application blueprint deployment diagram](assets/eab-deployment.svg) +![Enterprise Application blueprint deployment diagram](../assets/eab-deployment.svg) Each pipeline has the following associated resources: + - 2 Cloud Build triggers - 1 trigger to run Terraform Plan commands upon changes to a non-main git branch - 1 trigger to run Terraform Apply commands upon changes to the main git branch @@ -19,17 +20,30 @@ Each pipeline has the following associated resources: - Build Logs bucket, to store the logs from the build process - 1 service account for executing the Cloud Build build process - ## Usage ### Deploying with Cloud Build + #### Deploying on Enterprise Foundation blueprint + If you have previously deployed the Enterprise Foundation blueprint, create the pipelines in this phase by pushing the contents of this folder to a [workload repo created at stage 5](https://github.com/terraform-google-modules/terraform-example-foundation/blob/master/5-app-infra/README.md). Instead of deploying to multiple environments, create these pipelines in the common folder of the foundation. Start at "5. Clone the `bu1-example-app` repo". Replace the contents of that repo with the contents of this folder. ### Running Terraform locally +#### Requirements + +You will need a project to host your resources, you can manually create it: + +```txt +example-organization +└── fldr-common + └── prj-c-eab-bootstrap +``` + +#### Step-by-Step + 1. The next instructions assume that you are in the `terraform-google-enterprise-application/1-bootstrap` folder. ```bash @@ -42,7 +56,7 @@ Start at "5. Clone the `bu1-example-app` repo". Replace the contents of that rep mv terraform.example.tfvars terraform.tfvars ``` -1. Update the file with values for your environment. +1. Update the `terraform.tfvars` file with your project id. You can now deploy the common environment for these pipelines. diff --git a/1-bootstrap/terraform.example.tfvars b/1-bootstrap/terraform.example.tfvars new file mode 100644 index 00000000..a8c64695 --- /dev/null +++ b/1-bootstrap/terraform.example.tfvars @@ -0,0 +1 @@ +project_id = "REPLACE_WITH_YOUR_PROJECT" diff --git a/2-multitenant/README.md b/2-multitenant/README.md index ecee1b11..6887d68c 100644 --- a/2-multitenant/README.md +++ b/2-multitenant/README.md @@ -5,9 +5,10 @@ This phase deploys the per-environment multitenant resources deployed via the multitenant infrastructure pipeline. An overview of the multitenant infrastructure pipeline is shown below. -![Enterprise Application multitenant infrastructure diagram](assets/eab-multitenant.png) +![Enterprise Application multitenant infrastructure diagram](../assets/eab-multitenant.png) The following resources are created: + - GCP Project (cluster project) - GKE cluster(s) - Cloud Armor @@ -22,10 +23,10 @@ The following resources are created: ### Running Terraform locally -1. The next instructions assume that you are in the `terraform-google-enterprise-application/4-appfactory` folder. +1. The next instructions assume that you are in the `terraform-google-enterprise-application/2-multitenant` folder. ```bash - cd terraform-google-enterprise-application/4-appfactory + cd ../2-multitenant ``` 1. Rename `terraform.example.tfvars` to `terraform.tfvars`. @@ -40,7 +41,7 @@ on the values in the `terraform.tfvars` file. In addition to `envs` from prerequisites, each App must have it's own entry under `apps` with a list of any dedicated IP address to be provisioned. - ``` + ```terraform apps = { "my-app" : { "ip_address_names" : [ @@ -58,14 +59,18 @@ You can now deploy each of your environments (e.g. production). 1. Run `init` and `plan` and review the output. ```bash - terraform init -chdir=./envs/production - terraform plan -chdir=./envs/production + terraform -chdir=./envs/production init + terraform -chdir=./envs/production plan ``` 1. Run `apply production`. ```bash - terraform apply -chdir=./envs/production + terraform -chdir=./envs/production apply ``` -If you receive any errors or made any changes to the Terraform config or `terraform.tfvars`, re-run `terraform plan -chdir=./envs/production` before you run `terraform apply -chdir=./envs/production`. +If you receive any errors or made any changes to the Terraform config or `terraform.tfvars`, re-run `terraform -chdir=./envs/production plan` before you run `terraform -chdir=./envs/production apply`. + +1. Repeat the same series of terraform commands but replace `-chdir=./envs/production` with `-chdir=./envs/nonproduction` to deploy the nonproduction environment. + +1. Repeat the same series of terraform commands but replace `-chdir=./envs/production` with `-chdir=./envs/development` to deploy the development environment. diff --git a/3-fleetscope/README.md b/3-fleetscope/README.md index 3ad89779..001bd622 100644 --- a/3-fleetscope/README.md +++ b/3-fleetscope/README.md @@ -1,4 +1,5 @@ -# 4. Fleet Scope phase +# 3. Fleet Scope phase + The Fleet Scope phase defines the resources used to create the GKE Fleet Scopes, Fleet namespaces, and some Fleet features. ## Purpose @@ -6,9 +7,10 @@ The Fleet Scope phase defines the resources used to create the GKE Fleet Scopes, This phase deploys the per-environment fleet resources deployed via the fleetscope infrastructure pipeline. An overview of the fleet-scope pipeline is shown below. -![Enterprise Application fleet-scope diagram](assets/eab-multitenant.png) +![Enterprise Application fleet-scope diagram](../assets/eab-multitenant.png) The following resources are created: + - Fleet scope - Fleet namespace - Cloud Source Repo @@ -27,10 +29,10 @@ The following resources are created: ### Running Terraform locally -1. The next instructions assume that you are in the `terraform-google-enterprise-application/4-fleetscope` folder. +1. The next instructions assume that you are in the `terraform-google-enterprise-application/3-fleetscope` folder. ```bash - cd terraform-google-enterprise-application/4-fleetscope + cd ../3-fleetscope ``` 1. Rename `terraform.example.tfvars` to `terraform.tfvars`. @@ -46,14 +48,18 @@ You can now deploy each of your environments (e.g. production). 1. Run `init` and `plan` and review the output. ```bash - terraform init -chdir=./envs/production - terraform plan -chdir=./envs/production + terraform -chdir=./envs/production init + terraform -chdir=./envs/production plan ``` 1. Run `apply production`. ```bash - terraform apply -chdir=./envs/production + terraform -chdir=./envs/production apply ``` -If you receive any errors or made any changes to the Terraform config or `terraform.tfvars`, re-run `terraform plan -chdir=./envs/production` before you run `terraform apply -chdir=./envs/production`. +If you receive any errors or made any changes to the Terraform config or `terraform.tfvars`, re-run `terraform -chdir=./envs/production plan` before you run `terraform -chdir=./envs/production apply`. + +1. Repeat the same series of terraform commands but replace `-chdir=./envs/production` with `-chdir=./envs/nonproduction` to deploy the nonproduction environment. + +1. Repeat the same series of terraform commands but replace `-chdir=./envs/production` with `-chdir=./envs/development` to deploy the development environment. diff --git a/3-fleetscope/terraform.example.tfvars b/3-fleetscope/terraform.example.tfvars index d2eca10f..e3ccfe8a 100644 --- a/3-fleetscope/terraform.example.tfvars +++ b/3-fleetscope/terraform.example.tfvars @@ -14,9 +14,16 @@ * limitations under the License. */ +fleet_project_id = "{FLEET_PROJECT}" cluster_project_id = "{CLUSTER_PROJECT}" network_project_id = "{NETWORK_PROJECT}" cluster_membership_ids = [ "//gkehub.googleapis.com/projects/{CLUSTER_PROJECT}/locations/{REGION}/memberships/{MEMBERSHIP_ID}", "//gkehub.googleapis.com/projects/{CLUSTER_PROJECT}/locations/{REGION}/memberships/{MEMBERSHIP_ID}", ] + +namespace_ids = { + "frontend" = "your-frontend-group@yourdomain.com", + "accounts" = "your-accounts-group@yourdomain.com", + "transactions" = "your-transactions-group@yourdomain.com" +} diff --git a/4-appfactory/README.md b/4-appfactory/README.md index b7c6fd3c..6158f4e2 100644 --- a/4-appfactory/README.md +++ b/4-appfactory/README.md @@ -1,10 +1,11 @@ # 4. Application Factory phase ## Purpose + The application factory creates application project groups, which contain resources responsible for deployment of a single application within the developer platform. An overview of the application factory pipeline is shown below. -![Enterprise Application application factory diagram](assets/eab-app-factory.svg) +![Enterprise Application application factory diagram](../assets/eab-app-factory.svg) The application factory creates the following resources as defined in the [`app-group-baseline`](./modules/app-group-baseline/) submodule: @@ -13,6 +14,17 @@ The application factory creates the following resources as defined in the [`app- * **Infrastructure repository:** A Git repository containing the Terraform configuration for the application infrastructure. * **Application infrastucture pipeline:** A Cloud Build pipeline for deploying the application infrastructure specified as Terraform. +It will also create an Application Folder to group your admin projects under it, for example: + +```txt +. +└── fldr-common/ + ├── cymbal-bank/ + │ ├── accounts-userservice-admin + │ ├── accounts-contacts-admin + │ ├── ledger-ledger-writer-admin + │ └── ... +``` ## Usage @@ -21,7 +33,7 @@ The application factory creates the following resources as defined in the [`app- 1. The next instructions assume that you are in the `terraform-google-enterprise-application/4-appfactory` folder. ```bash - cd terraform-google-enterprise-application/4-appfactory + cd ../4-appfactory ``` 1. Rename `terraform.example.tfvars` to `terraform.tfvars`. @@ -32,19 +44,21 @@ The application factory creates the following resources as defined in the [`app- 1. Update the file with values for your environment. + > TIP: To retrieve the remote state bucket variable, you can run `terraform -chdir=../1-bootstrap/ output -raw state_bucket` command. + You can now deploy the into your common folder. 1. Run `init` and `plan` and review the output. ```bash - terraform init -chdir=./apps/cymbal-bank - terraform plan -chdir=./apps/cymbal-bank + terraform -chdir=./apps/cymbal-bank init + terraform -chdir=./apps/cymbal-bank plan ``` 1. Run `apply`. ```bash - terraform apply -chdir=./apps/cymbal-bank + terraform -chdir=./apps/cymbal-bank apply ``` -If you receive any errors or made any changes to the Terraform config or `terraform.tfvars`, re-run `terraform plan -chdir=./apps/cymbal-bank` before you run `terraform apply -chdir=./apps/cymbal-bank`. +If you receive any errors or made any changes to the Terraform config or `terraform.tfvars`, re-run `terraform -chdir=./apps/cymbal-bank plan` before you run `terraform -chdir=./apps/cymbal-bank apply`. diff --git a/4-appfactory/apps/cymbal-bank/README.md b/4-appfactory/apps/cymbal-bank/README.md index 07ee9772..9c3ef467 100644 --- a/4-appfactory/apps/cymbal-bank/README.md +++ b/4-appfactory/apps/cymbal-bank/README.md @@ -6,10 +6,11 @@ | billing\_account | Billing Account ID for application admin project resources. | `string` | n/a | yes | | bucket\_force\_destroy | When deleting a bucket, this boolean option will delete all contained objects. If false, Terraform will fail to delete buckets which contain objects. | `bool` | `false` | no | | bucket\_prefix | Name prefix to use for buckets created. | `string` | `"bkt"` | no | -| common\_folder\_id | Folder ID in which to create all application admin projects | `string` | n/a | yes | +| common\_folder\_id | Folder ID in which to create all application admin projects, must be prefixed with 'folders/' | `string` | n/a | yes | | envs | Environments | 
map(object({
    billing_account    = string
    folder_id          = string
    network_project_id = string
    network_self_link  = string
    org_id             = string
    subnets_self_links = list(string)
  }))
| n/a | yes | | location | Location for build buckets. | `string` | `"us-central1"` | no | | org\_id | Google Cloud Organization ID. | `string` | n/a | yes | +| remote\_state\_bucket | Backend bucket to load Terraform Remote State Data from previous steps. | `string` | n/a | yes | | tf\_apply\_branches | List of git branches configured to run terraform apply Cloud Build trigger. All other branches will run plan by default. | `list(string)` | 
[
  "development",
  "non\\-production",
  "production"
]
| no | | trigger\_location | Location of for Cloud Build triggers created in the workspace. If using private pools should be the same location as the pool. | `string` | `"global"` | no | @@ -17,6 +18,7 @@ | Name | Description | |------|-------------| +| app-folders-ids | Pair of app-name and folder\_id | | app-group | Description on the app-group components | diff --git a/4-appfactory/apps/cymbal-bank/iam.tf b/4-appfactory/apps/cymbal-bank/iam.tf new file mode 100644 index 00000000..f62f1c01 --- /dev/null +++ b/4-appfactory/apps/cymbal-bank/iam.tf @@ -0,0 +1,41 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +locals { + all_environments_cluster_service_accounts_iam_members = [for sa in local.cluster_service_accounts : "serviceAccount:${sa}"] + + expanded_cluster_service_accounts = flatten([ + for key in keys(local.app_services) : [ + for sa in local.all_environments_cluster_service_accounts_iam_members : { + app_name = key + cluster_sa_member = sa + } + ] + ]) +} + +// Assign artifactregistry reader to cluster service accounts +// This allows docker images on application projects to be downloaded on the cluster +resource "google_folder_iam_member" "admin" { + // for each app folder, create permissions for dev/non prod and prod cluster service accounts + for_each = tomap({ + for app_name_sa in local.expanded_cluster_service_accounts : "${app_name_sa.app_name}.${app_name_sa.cluster_sa_member}" => app_name_sa + }) + + folder = google_folder.app_folder[each.value.app_name].name + role = "roles/artifactregistry.reader" + member = each.value.cluster_sa_member +} diff --git a/4-appfactory/apps/cymbal-bank/main.tf b/4-appfactory/apps/cymbal-bank/main.tf index 27274b82..db7f5848 100644 --- a/4-appfactory/apps/cymbal-bank/main.tf +++ b/4-appfactory/apps/cymbal-bank/main.tf @@ -15,26 +15,47 @@ */ locals { - components = [ - "balancereader", - "contacts", - "frontend", - "ledgerwriter", - "transactionhistory", - "userservice", - ] + app_services = { + "cymbal-bank" = [ + "balancereader", + "contacts", + "frontend", + "ledgerwriter", + "transactionhistory", + "userservice", + ] + } + + expanded_app_services = flatten([ + for key, services in local.app_services : [ + for service in services : { + app_name = key + service_name = service + } + ] + ]) +} + +// One folder per application, will group admin/service projects under it +resource "google_folder" "app_folder" { + for_each = local.app_services + + display_name = each.key + parent = var.common_folder_id } module "components" { - for_each = toset(local.components) - source = "../../modules/app-group-baseline" + for_each = tomap({ + for app_service in local.expanded_app_services : "${app_service.app_name}.${app_service.service_name}" => app_service + }) + source = "../../modules/app-group-baseline" - application_name = each.value + application_name = each.value.service_name create_env_projects = true org_id = var.org_id billing_account = var.billing_account - folder_id = var.common_folder_id + folder_id = google_folder.app_folder[each.value.app_name].folder_id envs = var.envs bucket_prefix = var.bucket_prefix location = var.location diff --git a/4-appfactory/apps/cymbal-bank/outputs.tf b/4-appfactory/apps/cymbal-bank/outputs.tf index 4269093b..9d6c7e3c 100644 --- a/4-appfactory/apps/cymbal-bank/outputs.tf +++ b/4-appfactory/apps/cymbal-bank/outputs.tf @@ -30,3 +30,10 @@ output "app-group" { } } } + +output "app-folders-ids" { + description = "Pair of app-name and folder_id" + value = { + for k, v in google_folder.app_folder : k => v.folder_id + } +} diff --git a/4-appfactory/apps/cymbal-bank/remote.tf b/4-appfactory/apps/cymbal-bank/remote.tf new file mode 100644 index 00000000..c40474bb --- /dev/null +++ b/4-appfactory/apps/cymbal-bank/remote.tf @@ -0,0 +1,32 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +// These values are retrieved from the saved terraform state of the execution +// of previous step using the terraform_remote_state data source. +locals { + cluster_service_accounts = flatten([for state in data.terraform_remote_state.multitenant : state.outputs.cluster_service_accounts]) +} + +data "terraform_remote_state" "multitenant" { + for_each = var.envs + + backend = "gcs" + + config = { + bucket = var.remote_state_bucket + prefix = "terraform/multi_tenant/${each.key}" + } +} diff --git a/4-appfactory/apps/cymbal-bank/terraform.tfvars b/4-appfactory/apps/cymbal-bank/terraform.tfvars new file mode 120000 index 00000000..00f38576 --- /dev/null +++ b/4-appfactory/apps/cymbal-bank/terraform.tfvars @@ -0,0 +1 @@ +../../terraform.tfvars \ No newline at end of file diff --git a/4-appfactory/apps/cymbal-bank/variables.tf b/4-appfactory/apps/cymbal-bank/variables.tf index 967c6d5c..26af219a 100644 --- a/4-appfactory/apps/cymbal-bank/variables.tf +++ b/4-appfactory/apps/cymbal-bank/variables.tf @@ -16,7 +16,12 @@ variable "common_folder_id" { type = string - description = "Folder ID in which to create all application admin projects" + description = "Folder ID in which to create all application admin projects, must be prefixed with 'folders/'" + + validation { + condition = can(regex("^folders/", var.common_folder_id)) + error_message = "The folder ID must be prefixed with 'folders/'." + } } variable "org_id" { @@ -70,3 +75,8 @@ variable "tf_apply_branches" { type = list(string) default = ["development", "non\\-production", "production"] } + +variable "remote_state_bucket" { + description = "Backend bucket to load Terraform Remote State Data from previous steps." + type = string +} diff --git a/4-appfactory/apps/cymbal-bank/versions.tf b/4-appfactory/apps/cymbal-bank/versions.tf index ab78a33b..dce3110d 100644 --- a/4-appfactory/apps/cymbal-bank/versions.tf +++ b/4-appfactory/apps/cymbal-bank/versions.tf @@ -17,7 +17,14 @@ terraform { required_version = ">= 1.3" + required_providers { + google = { + source = "hashicorp/google" + version = ">= 5, < 7" + } + } + provider_meta "google" { - module_name = "blueprints/terraform/terraform-google-enterprise-application:bootstrap/v0.1.0" + module_name = "blueprints/terraform/terraform-google-enterprise-application:appfactory/cymbal-bank/v0.1.0" } } diff --git a/4-appfactory/terraform.example.tfvars b/4-appfactory/terraform.example.tfvars new file mode 100644 index 00000000..60065e1c --- /dev/null +++ b/4-appfactory/terraform.example.tfvars @@ -0,0 +1,54 @@ +/** + * Copyright 2024 Google LLC + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +billing_account = "REPLACE_WITH_BILLING_ACCOUNT" +common_folder_id = "REPLACE_WITH_COMMON_FOLDER_ID" +org_id = "REPLACE_WITH_YOUR_ORGANIZATION_ID" +remote_state_bucket = "REPLACE_WITH_YOUR_REMOTE_STATE_BUCKET" +envs = { + "development" = { + "billing_account" = "{BILLING_ACCOUNT}" + "folder_id" = "folders/{FOLDER}" + "network_project_id" = "{PROJECT}" + "network_self_link" = "https://www.googleapis.com/compute/v1/projects/{PROJECT}/global/networks/{NETWORK}}" + "org_id" = "{ORGANIZATION}" + "subnets_self_links" = [ + "https://www.googleapis.com/compute/v1/projects/{PROJECT}/regions/{REGION}/subnetworks/{SUBNETWORK}", + ] + } + "non-production" = { + "billing_account" = "{BILLING_ACCOUNT}" + "folder_id" = "folders/{FOLDER}" + "network_project_id" = "{PROJECT}" + "network_self_link" = "https://www.googleapis.com/compute/v1/projects/{PROJECT}/global/networks/{NETWORK}}" + "org_id" = "{ORGANIZATION}" + "subnets_self_links" = [ + "https://www.googleapis.com/compute/v1/projects/{PROJECT}/regions/{REGION}/subnetworks/{SUBNETWORK}", + "https://www.googleapis.com/compute/v1/projects/{PROJECT}/regions/{REGION}/subnetworks/{SUBNETWORK}", + ] + } + "production" = { + "billing_account" = "{BILLING_ACCOUNT}" + "folder_id" = "folders/{FOLDER}" + "network_project_id" = "{PROJECT}" + "network_self_link" = "https://www.googleapis.com/compute/v1/projects/{PROJECT}/global/networks/{NETWORK}}" + "org_id" = "{ORGANIZATION}" + "subnets_self_links" = [ + "https://www.googleapis.com/compute/v1/projects/{PROJECT}/regions/{REGION}/subnetworks/{SUBNETWORK}", + "https://www.googleapis.com/compute/v1/projects/{PROJECT}/regions/{REGION}/subnetworks/{SUBNETWORK}", + ] + } +} diff --git a/5-appinfra/README.md b/5-appinfra/README.md index 090ff39c..3f9d7b0c 100644 --- a/5-appinfra/README.md +++ b/5-appinfra/README.md @@ -1,12 +1,14 @@ # 5. Application Infrastructure pipeline ## Purpose + The application infrastructure pipeline deploys the application-specific CI/CD resources as well as any additional application-specific services, such as databases or other managed services. An overview of application inrastruction pipeline is shown below, in the context of deploying a new applicaiton across the Enterprise Application blueprint. -![Enterprise Application application infrastructure diagram](assets/eab-app-deployment.svg) +![Enterprise Application application infrastructure diagram](../assets/eab-app-deployment.svg) ### Application CI/CD pipeline + The application infrastructure pipeline creates the following resources to establish the application CI/CD pipeline, as defined in the [`cicd-pipeline`](./modules/cicd-pipeline/) submodule: - Cloud Build trigger @@ -32,6 +34,7 @@ You may add additional infrastructure like application-specifc databases or othe ```bash cd terraform-google-enterprise-application/5-appinfra ``` + Under the `apps` folder are examples for each of the cymbal bank applications. 1. Change directory into any of these folders to deploy. @@ -47,14 +50,14 @@ Deploy the `shared` environment first, which contains the application CI/CD pipe 1. Run `init` and `plan` and review the output. ```bash - terraform init -chdir=./envs/shared - terraform plan -chdir=./envs/shared + terraform -chdir=./envs/shared init + terraform -chdir=./envs/shared plan ``` 1. Run `apply shared`. ```bash - terraform apply -chdir=./envs/shared + terraform -chdir=./envs/shared apply ``` You can now deploy each of your environments (e.g. production). @@ -62,14 +65,14 @@ You can now deploy each of your environments (e.g. production). 1. Run `init` and `plan` and review the output. ```bash - terraform init -chdir=./envs/production - terraform plan -chdir=./envs/production + terraform -chdir=./envs/production init + terraform -chdir=./envs/production plan ``` 1. Run `apply production`. ```bash - terraform apply -chdir=./envs/production + terraform -chdir=./envs/production apply ``` -If you receive any errors or made any changes to the Terraform config or `terraform.tfvars`, re-run `terraform plan -chdir=./envs/production` before you run `terraform apply -chdir=./envs/production`. +If you receive any errors or made any changes to the Terraform config or `terraform.tfvars`, re-run `terraform -chdir=./envs/production plan` before you run `terraform apply -chdir=./envs/production`. diff --git a/5-appinfra/modules/cicd-pipeline/artifact-registry.tf b/5-appinfra/modules/cicd-pipeline/artifact-registry.tf index b68ccd5d..4b628bc7 100644 --- a/5-appinfra/modules/cicd-pipeline/artifact-registry.tf +++ b/5-appinfra/modules/cicd-pipeline/artifact-registry.tf @@ -25,22 +25,18 @@ resource "google_artifact_registry_repository" "container_registry" { ] } -module "artifact-registry-repository-iam-bindings" { - source = "terraform-google-modules/iam/google//modules/artifact_registry_iam" - version = "~> 7.7" - project = var.project_id - repositories = [local.service_name] - location = var.region - mode = "authoritative" - - bindings = { - "roles/artifactregistry.reader" = [ - "serviceAccount:${data.google_project.project.number}-compute@developer.gserviceaccount.com", - "serviceAccount:${google_service_account.cloud_deploy.email}", - "allAuthenticatedUsers" - ], +resource "google_artifact_registry_repository_iam_member" "member" { + for_each = { + "compute" = "serviceAccount:${data.google_project.project.number}-compute@developer.gserviceaccount.com", + "cloud_deploy" = "serviceAccount:${google_service_account.cloud_deploy.email}", } + project = var.project_id + location = var.region + repository = local.service_name + role = "roles/artifactregistry.reader" + member = each.value + depends_on = [ module.enabled_google_apis, google_artifact_registry_repository.container_registry diff --git a/test/integration/appfactory/appfactory_test.go b/test/integration/appfactory/appfactory_test.go index 9631a58e..785fb184 100644 --- a/test/integration/appfactory/appfactory_test.go +++ b/test/integration/appfactory/appfactory_test.go @@ -16,6 +16,7 @@ package appfactory import ( "fmt" + "strings" "testing" "time" @@ -41,6 +42,7 @@ func TestAppfactory(t *testing.T) { } vars := map[string]interface{}{ + "remote_state_bucket": backend_bucket, "bucket_force_destroy": "true", } @@ -59,13 +61,50 @@ func TestAppfactory(t *testing.T) { appFactory.DefineVerify(func(assert *assert.Assertions) { appFactory.DefaultVerify(assert) + // retrieve all cluster service accounts from all multitenant environments + var allClusterServiceAccounts []string + + for _, envName := range testutils.EnvNames { + multitenant := tft.NewTFBlueprintTest(t, + tft.WithTFDir(fmt.Sprintf("../../../2-multitenant/envs/%s", envName)), + ) + // add to slice the environment service accounts + for _, sa := range multitenant.GetJsonOutput("cluster_service_accounts").Array() { + allClusterServiceAccounts = append(allClusterServiceAccounts, ("serviceAccount:" + sa.String())) + } + } + + assert.Greater(len(allClusterServiceAccounts), 0, "The slice of cluster service accounts must contain more than 0 service accounts.") + + // check if created folders contain artifactregistry.reader for the cluster service accounts + // this is necessary to ensure the cluster can download docker images + for _, folderId := range appFactory.GetJsonOutput("app-folders-ids").Map() { + t.Run(folderId.String(), func(t *testing.T) { + t.Parallel() + folderIamPolicy := gcloud.Runf(t, "resource-manager folders get-iam-policy %s", folderId.String()) + // ensure cluster sa is in folder iam policy for artifactregistry.reader role + for _, binding := range folderIamPolicy.Get("bindings").Array() { + if binding.Get("role").String() == "roles/artifactregistry.reader" { + folderIamPolicyMembers := binding.Get("members").Array() + for _, sa := range allClusterServiceAccounts { + assert.True(testutils.Contains(folderIamPolicyMembers, sa), fmt.Sprintf("The cluster service account %s must exist in the folder %s artifactregistry.reader iam policy", sa, folderId)) + } + } + } + }) + } + // check admin projects // TODO: Update to use https://github.com/GoogleCloudPlatform/cloud-foundation-toolkit/pull/2356 when released. // terraform.OutputJson OK to use as long as there is only one appGroupName - for appName, appData := range gjson.Parse(terraform.OutputJson(t, appFactory.GetTFOptions(), "app-group")).Map() { - appName := appName + for applicationService, appData := range gjson.Parse(terraform.OutputJson(t, appFactory.GetTFOptions(), "app-group")).Map() { + parts := strings.Split(applicationService, ".") + assert.Equal(len(parts), 2, "The keys of app-group output must be in the format 'app-name'.'service-name', for example: 'cymbal-bank.userservice'") + appName := parts[0] + serviceName := parts[1] + appData := appData - t.Run(appName, func(t *testing.T) { + t.Run(fmt.Sprintf("%s.%s", appName, serviceName), func(t *testing.T) { t.Parallel() adminProjectID := appData.Get("app_admin_project_id").String() @@ -126,7 +165,7 @@ func TestAppfactory(t *testing.T) { }, } { bucketSelfLink := appData.Get(bucket.output).String() - opBucket := gcloud.Run(t, fmt.Sprintf("storage ls --buckets gs://%s-%s-%s-%s", bucket.prefix, adminProjectID, appName, bucket.suffix), gcloudArgsBucket).Array() + opBucket := gcloud.Run(t, fmt.Sprintf("storage ls --buckets gs://%s-%s-%s-%s", bucket.prefix, adminProjectID, serviceName, bucket.suffix), gcloudArgsBucket).Array() assert.Equal(bucketSelfLink, opBucket[0].Get("metadata.selfLink").String(), fmt.Sprintf("The bucket SelfLink should be %s.", bucketSelfLink)) } // triggers diff --git a/test/integration/appinfra/appinfra_test.go b/test/integration/appinfra/appinfra_test.go index 38518cee..d82e306d 100644 --- a/test/integration/appinfra/appinfra_test.go +++ b/test/integration/appinfra/appinfra_test.go @@ -33,7 +33,7 @@ func TestAppInfra(t *testing.T) { env_cluster_membership_ids := make(map[string]map[string][]string, 0) for _, envName := range testutils.EnvNames { env_cluster_membership_ids[envName] = make(map[string][]string, 0) - multitenant := tft.NewTFBlueprintTest(t, tft.WithTFDir(fmt.Sprintf("../../../2-multitenant/envs/%s", envName))) + multitenant := tft.NewTFBlueprintTest(t, tft.WithTFDir(fmt.Sprintf("../../../2-multitenant/envs/%s", envName))) env_cluster_membership_ids[envName]["cluster_membership_ids"] = testutils.GetBptOutputStrSlice(multitenant, "cluster_membership_ids") } @@ -60,7 +60,7 @@ func TestAppInfra(t *testing.T) { splitServiceName = strings.Split(fullServiceName, "-") prefixServiceName = splitServiceName[0] suffixServiceName = splitServiceName[len(splitServiceName)-1] - projectID := appFactory.GetJsonOutput("app-group").Get(fmt.Sprintf("%s.app_admin_project_id", suffixServiceName)).String() + projectID := appFactory.GetJsonOutput("app-group").Get(fmt.Sprintf("%s\\.%s.app_admin_project_id", appName, suffixServiceName)).String() servicesInfoMap[fullServiceName] = ServiceInfos{ ApplicationName: appName, ProjectID: projectID, @@ -73,10 +73,10 @@ func TestAppInfra(t *testing.T) { t.Parallel() vars := map[string]interface{}{ - "project_id": servicesInfoMap[fullServiceName].ProjectID, - "region": region, - "env_cluster_membership_ids": env_cluster_membership_ids, - "buckets_force_destroy": "true", + "project_id": servicesInfoMap[fullServiceName].ProjectID, + "region": region, + "env_cluster_membership_ids": env_cluster_membership_ids, + "buckets_force_destroy": "true", } appService := tft.NewTFBlueprintTest(t, @@ -120,7 +120,6 @@ func TestAppInfra(t *testing.T) { arRegistryIAMMembers := []string{ fmt.Sprintf("serviceAccount:%s-compute@developer.gserviceaccount.com", projectNumber), fmt.Sprintf("serviceAccount:deploy-%s@%s.iam.gserviceaccount.com", servicesInfoMap[fullServiceName].ServiceName, servicesInfoMap[fullServiceName].ProjectID), - "allAuthenticatedUsers", } arRegistrySAIamFilter := "bindings.role:'roles/artifactregistry.reader'" arRegistrySAIamCommonArgs := gcloud.WithCommonArgs([]string{"--flatten", "bindings", "--filter", arRegistrySAIamFilter, "--format", "json"}) @@ -153,7 +152,7 @@ func TestAppInfra(t *testing.T) { } for env := range env_cluster_membership_ids { - bucketName :=fmt.Sprintf("artifacts-%s-%s-%s", env, projectNumber, servicesInfoMap[fullServiceName].ServiceName) + bucketName := fmt.Sprintf("artifacts-%s-%s-%s", env, projectNumber, servicesInfoMap[fullServiceName].ServiceName) bucketOp := gcloud.Runf(t, "storage buckets describe gs://%s --project %s", bucketName, servicesInfoMap[fullServiceName].ProjectID) assert.True(bucketOp.Get("uniform_bucket_level_access").Bool(), fmt.Sprintf("Bucket %s should have uniform access level.", bucketName)) diff --git a/test/integration/appsource/cymbal_bank_test.go b/test/integration/appsource/cymbal_bank_test.go index 3ec6f1a2..0d667905 100644 --- a/test/integration/appsource/cymbal_bank_test.go +++ b/test/integration/appsource/cymbal_bank_test.go @@ -64,7 +64,7 @@ func TestSourceCymbalBank(t *testing.T) { splitServiceName = strings.Split(serviceName, "-") prefixServiceName = splitServiceName[0] suffixServiceName = splitServiceName[len(splitServiceName)-1] - projectID := appFactory.GetJsonOutput("app-group").Get(fmt.Sprintf("%s.app_admin_project_id", suffixServiceName)).String() + projectID := appFactory.GetJsonOutput("app-group").Get(fmt.Sprintf("%s\\.%s.app_admin_project_id", appName, suffixServiceName)).String() servicesInfoMap[serviceName] = ServiceInfos{ ProjectID: projectID, ServiceName: suffixServiceName, diff --git a/test/integration/testutils/utils.go b/test/integration/testutils/utils.go index 7fca92aa..6338f3a1 100644 --- a/test/integration/testutils/utils.go +++ b/test/integration/testutils/utils.go @@ -44,3 +44,13 @@ func Filter(field string, value string, iamList []gjson.Result) []gjson.Result { } return filtered } + +// verify if gjson array of string contains another string +func Contains(slice []gjson.Result, item string) bool { + for _, v := range slice { + if v.String() == item { + return true + } + } + return false +}