This blueprint creates a Private instance of Cloud Composer version 2 on a VPC with a dedicated service account. Cloud Composer 2 is the new major version for Cloud Composer that supports:
- environment autoscaling
- workloads configuration: CPU, memory, and storage parameters for Airflow workers, schedulers, web server, and database.
Please consult the documentation page for an exhaustive comparison between Composer Version 1 and Version 2.
The solution will use:
- Cloud Composer
- VPC with Private Service Access to deploy resources, if no Shared VPC configuration provided.
- Google Cloud NAT to access internet resources, if no Shared VPC configuration provided.
The solution supports as inputs:
- Shared VPC
- Cloud KMS CMEK keys
This is the high level diagram:
This blueprint will deploy all its resources into the project defined by the project_id variable. Please note that we assume this project already exists. However, if you provide the appropriate values to the project_create
variable, the project will be created as part of the deployment.
If project_create
is left to null, the identity performing the deployment needs the owner role on the project defined by the project_id
variable. Otherwise, the identity performing the deployment needs resourcemanager.projectCreator
on the resource hierarchy node specified by project_create.parent
and billing.user
on the billing account specified by project_create.billing_account_id
.
Run Terraform init:
terraform init
Configure the Terraform variable in your terraform.tfvars file. You need to specify at least the following variables:
project_id = "lcaggioni-sandbox"
prefix = "lc"
region = "europe-west1"
You can run now:
terraform apply
You can now connect to your instance.
If a shared VPC is not configured, a VPC will be created within the project. The following IP ranges will be used:
- Cloudsql:
10.20.10.0/24
- GKE:
10.20.11.0/28
Change the code as needed to match your needed configuration, remember that these addresses should not overlap with any other range used in network.
As is often the case in real-world configurations, this blueprint accepts as input an existing Shared-VPC
via the network_config
variable.
Example:
network_config = {
host_project = "PROJECT"
network_self_link = "projects/PROJECT/global/networks/VPC_NAME"
region = "europe-west3"
subnet_self_link = "projects/PROJECT/regions/REGION/subnetworks/VPC_NAME"
composer_secondary_ranges = {
pods = "pods"
services = "services"
}
}
# tftest skip
Make sure that:
- The GKE API (
container.googleapis.com
) is enabled in the VPC host project. - The subnet has secondary ranges configured with 2 ranges:
- pods:
/22
example:10.10.8.0/22
- services =
/24
example: 10.10.12.0/24`
- pods:
- Firewall rules are set, as described in the documentation
In order to run the example and deploy Cloud Composer on a shared VPC the identity running Terraform must have the following IAM role on the Shared VPC Host project.
- Compute Network Admin (roles/compute.networkAdmin)
- Compute Shared VPC Admin (roles/compute.xpnAdmin)
As is often the case in real-world configurations, this blueprint accepts as input an existing Cloud KMS keys
via the service_encryption_keys
variable.
Example:
service_encryption_keys = {
`europe/west1` = `projects/PROJECT/locations/REGION/keyRings/KR_NAME/cryptoKeys/KEY_NAME`
}
# tftest skip
name | description | type | required | default |
---|---|---|---|---|
prefix | Prefix used for resource names. | string |
✓ | |
project_id | Project id, references existing project if project_create is null. |
string |
✓ | |
region | Region where instances will be deployed. | string |
✓ | |
composer_config | Composer environment configuration. It accepts only following attributes: environment_size , software_config and workloads_config . See attribute reference for details on settings variables. |
object({…}) |
{…} |
|
iam_bindings_additive | Map of Role => principal in IAM format (group:[email protected] ) to be added on the project. |
map(list(string)) |
{} |
|
network_config | Shared VPC network configurations to use. If null networks will be created in projects with preconfigured values. | object({…}) |
null |
|
project_create | Provide values if project creation is needed, uses existing project if null. Parent is in 'folders/nnn' or 'organizations/nnn' format. | object({…}) |
null |
|
service_encryption_keys | Cloud KMS keys to use to encrypt resources. Provide a key for each region in use. | map(string) |
null |
name | description | sensitive |
---|---|---|
composer_airflow_uri | The URI of the Apache Airflow Web UI hosted within the Cloud Composer environment.. | |
composer_dag_gcs | The Cloud Storage prefix of the DAGs for the Cloud Composer environment. | |
composer_service_account | Cloud Composer nodes Service Account email. |
module "test" {
source = "./fabric/blueprints/data-solutions/composer-2/"
project_id = "composer"
project_create = {
billing_account_id = var.billing_account_id
parent = var.folder_id
}
prefix = var.prefix
region = var.region
}
# tftest modules=5 resources=29 e2e