Factories refactor

.github/workflows/linting.yml

@@ -34,7 +34,7 @@ jobs:
       - name: Set up Terraform
         uses: hashicorp/setup-terraform@v2
         with:
-          terraform_version: 1.7.0
+          terraform_version: 1.7.4
 
       - name: Install dependencies
         run: |

.github/workflows/tests.yml

@@ -25,7 +25,7 @@ env:
   PYTEST_ADDOPTS: "--color=yes"
   PYTHON_VERSION: "3.10"
   TF_PLUGIN_CACHE_DIR: "/home/runner/.terraform.d/plugin-cache"
-  TF_VERSION: 1.7.0
+  TF_VERSION: 1.7.4
   TFTEST_COPY: 1
 
 jobs:

CHANGELOG.md

@@ -1653,7 +1653,7 @@ All notable changes to this project will be documented in this file.
 - add support for VPC-SC perimeters in Data Foundation end to end example
 - fix `vpc-sc` module
 - new networking example showing how to use [Private Service Connect to call a Cloud Function from on-premises](./blueprints/networking/private-cloud-function-from-onprem/)
-- new networking example showing how to organize [decentralized firewall](./blueprints/networking/decentralized-firewall/) management on GCP
+- new networking example showing how to organize decentralized firewall management on GCP
 
 ## [5.0.0] - 2021-06-17
 

README.md

@@ -30,6 +30,7 @@ The current list of modules supports most of the core foundational and networkin
 Currently available modules:
 
 - **foundational** - [billing account](./modules/billing-account), [Cloud Identity group](./modules/cloud-identity-group/), [folder](./modules/folder), [service accounts](./modules/iam-service-account), [logging bucket](./modules/logging-bucket), [organization](./modules/organization), [project](./modules/project), [projects-data-source](./modules/projects-data-source)
+- **process factories** - [project factory](./modules/project-factory/README.md)
 - **networking** - [DNS](./modules/dns), [DNS Response Policy](./modules/dns-response-policy/), [Cloud Endpoints](./modules/endpoints), [address reservation](./modules/net-address), [NAT](./modules/net-cloudnat), [VLAN Attachment](./modules/net-vlan-attachment/), [External Application LB](./modules/net-lb-app-ext/), [External Passthrough Network LB](./modules/net-lb-ext), [External Regional Application Load Balancer](./modules/net-lb-app-ext-regional/), [Firewall policy](./modules/net-firewall-policy), [Internal Application LB](./modules/net-lb-app-int), [Cross-region Internal Application LB](./modules/net-lb-app-int-cross-region), [Internal Passthrough Network LB](./modules/net-lb-int), [Internal Proxy Network LB](./modules/net-lb-proxy-int), [IPSec over Interconnect](./modules/net-ipsec-over-interconnect), [VPC](./modules/net-vpc), [VPC firewall](./modules/net-vpc-firewall), [VPC peering](./modules/net-vpc-peering), [VPN dynamic](./modules/net-vpn-dynamic), [HA VPN](./modules/net-vpn-ha), [VPN static](./modules/net-vpn-static), [Service Directory](./modules/service-directory), [Secure Web Proxy](./modules/net-swp)
 - **compute** - [VM/VM group](./modules/compute-vm), [MIG](./modules/compute-mig), [COS container](./modules/cloud-config-container/cos-generic-metadata/) (coredns, mysql, onprem, squid), [GKE cluster](./modules/gke-cluster-standard), [GKE hub](./modules/gke-hub), [GKE nodepool](./modules/gke-nodepool), [GCVE private cloud](./modules/gcve-private-cloud)
 - **data** - <!-- [AlloyDB instance](./modules/alloydb-instance), --> [Analytics Hub](./modules/analytics-hub), [BigQuery dataset](./modules/bigquery-dataset), [Bigtable instance](./modules/bigtable-instance), [Dataplex](./modules/dataplex), [Dataplex DataScan](./modules/dataplex-datascan/), [Cloud SQL instance](./modules/cloudsql-instance), [Data Catalog Policy Tag](./modules/data-catalog-policy-tag), [Data Catalog Tag](./modules/data-catalog-tag), [Data Catalog Tag Template](./modules/data-catalog-tag-template), [Datafusion](./modules/datafusion), [Dataproc](./modules/dataproc), [GCS](./modules/gcs), [Pub/Sub](./modules/pubsub), [Dataform Repository](./modules/dataform-repository/)

blueprints/README.md

@@ -7,9 +7,9 @@ Currently available blueprints:
 - **apigee** - [Apigee Hybrid on GKE](./apigee/hybrid-gke/), [Apigee X analytics in BigQuery](./apigee/bigquery-analytics), [Apigee network patterns](./apigee/network-patterns/)
 - **cloud operations** - [Active Directory Federation Services](./cloud-operations/adfs), [Cloud Asset Inventory feeds for resource change tracking and remediation](./cloud-operations/asset-inventory-feed-remediation), [Fine-grained Cloud DNS IAM via Service Directory](./cloud-operations/dns-fine-grained-iam), [Cloud DNS & Shared VPC design](./cloud-operations/dns-shared-vpc), [Delegated Role Grants](./cloud-operations/iam-delegated-role-grants), [Network Quota Monitoring](./cloud-operations/network-quota-monitoring), [Managing on-prem service account keys by uploading public keys](./cloud-operations/onprem-sa-key-management), [Compute Image builder with Hashicorp Packer](./cloud-operations/packer-image-builder), [Packer example](./cloud-operations/packer-image-builder/packer), [Compute Engine quota monitoring](./cloud-operations/compute-quota-monitoring), [Scheduled Cloud Asset Inventory Export to Bigquery](./cloud-operations/scheduled-asset-inventory-export-bq), [Configuring workload identity federation with Terraform Cloud/Enterprise workflows](./cloud-operations/terraform-cloud-dynamic-credentials), [TCP healthcheck and restart for unmanaged GCE instances](./cloud-operations/unmanaged-instances-healthcheck), [Migrate for Compute Engine (v5) blueprints](./cloud-operations/vm-migration), [Configuring workload identity federation to access Google Cloud resources from apps running on Azure](./cloud-operations/workload-identity-federation)
 - **data solutions** - [GCE and GCS CMEK via centralized Cloud KMS](./data-solutions/cmek-via-centralized-kms), [Cloud Composer version 2 private instance, supporting Shared VPC and external CMEK key](./data-solutions/composer-2), [Cloud SQL instance with multi-region read replicas](./data-solutions/cloudsql-multiregion), [Data Platform](./data-solutions/data-platform-foundations), [Minimal Data Platform](./data-solutions/data-platform-minimal), [Spinning up a foundation data pipeline on Google Cloud using Cloud Storage, Dataflow and BigQuery](./data-solutions/gcs-to-bq-with-least-privileges), [#SQL Server Always On Groups blueprint](./data-solutions/sqlserver-alwayson), [Data Playground](./data-solutions/data-playground), [MLOps with Vertex AI](./data-solutions/vertex-mlops), [Shielded Folder](./data-solutions/shielded-folder), [BigQuery ML and Vertex AI Pipeline](./data-solutions/bq-ml)
-- **factories** - [The why and the how of Resource Factories](./factories), [Google Cloud Identity Group Factory](./factories/cloud-identity-group-factory), [Google Cloud BQ Factory](./factories/bigquery-factory), [Google Cloud VPC Firewall Factory](./factories/net-vpc-firewall-yaml), [Minimal Project Factory](./factories/project-factory)
+- **factories** - [Fabric resource factories](./factories)
 - **GKE** - [Binary Authorization Pipeline Blueprint](./gke/binauthz), [Storage API](./gke/binauthz/image), [Multi-cluster mesh on GKE (fleet API)](./gke/multi-cluster-mesh-gke-fleet-api), [GKE Multitenant Blueprint](./gke/multitenant-fleet), [Shared VPC with GKE support](./networking/shared-vpc-gke/), [GKE Autopilot](./gke/autopilot)
-- **networking** - [Calling a private Cloud Function from On-premises](./networking/private-cloud-function-from-onprem), [Decentralized firewall management](./networking/decentralized-firewall), [Decentralized firewall validator](./networking/decentralized-firewall/validator), [HA VPN over Interconnect](./networking/ha-vpn-over-interconnect/), [GLB and multi-regional daisy-chaining through hybrid NEGs](./networking/glb-hybrid-neg-internal), [Hybrid connectivity to on-premise services through PSC](./networking/psc-hybrid), [HTTP Load Balancer with Cloud Armor](./networking/glb-and-armor), [Hub and Spoke via VPN](./networking/hub-and-spoke-vpn), [Hub and Spoke via VPC Peering](./networking/hub-and-spoke-peering), [Internal Load Balancer as Next Hop](./networking/ilb-next-hop), On-prem DNS and Google Private Access,  [PSC Producer](./networking/psc-hybrid/psc-producer), [PSC Consumer](./networking/psc-hybrid/psc-consumer), [Shared VPC with optional GKE cluster](./networking/shared-vpc-gke), [VPC Connectivity Lab](./networking/vpc-connectivity-lab/)
+- **networking** - [Calling a private Cloud Function from On-premises](./networking/private-cloud-function-from-onprem), [HA VPN over Interconnect](./networking/ha-vpn-over-interconnect/), [GLB and multi-regional daisy-chaining through hybrid NEGs](./networking/glb-hybrid-neg-internal), [Hybrid connectivity to on-premise services through PSC](./networking/psc-hybrid), [HTTP Load Balancer with Cloud Armor](./networking/glb-and-armor), [Hub and Spoke via VPN](./networking/hub-and-spoke-vpn), [Hub and Spoke via VPC Peering](./networking/hub-and-spoke-peering), [Internal Load Balancer as Next Hop](./networking/ilb-next-hop), On-prem DNS and Google Private Access,  [PSC Producer](./networking/psc-hybrid/psc-producer), [PSC Consumer](./networking/psc-hybrid/psc-consumer), [Shared VPC with optional GKE cluster](./networking/shared-vpc-gke), [VPC Connectivity Lab](./networking/vpc-connectivity-lab/)
 - **serverless** - [Cloud Run series](./serverless/cloud-run-explore)
 - **third party solutions** - [OpenShift on GCP user-provisioned infrastructure](./third-party-solutions/openshift), [Wordpress deployment on Cloud Run](./third-party-solutions/wordpress/cloudrun)
 

blueprints/data-solutions/shielded-folder/main.tf

@@ -94,7 +94,7 @@ module "firewall-policy" {
   source    = "../../../modules/net-firewall-policy"
   name      = "default"
   parent_id = module.folder.id
-  rules_factory_config = var.data_dir == null ? {} : {
+  factories_config = var.data_dir == null ? {} : {
     cidr_file_path          = "${var.data_dir}/firewall-policies/cidrs.yaml"
     ingress_rules_file_path = "${var.data_dir}/firewall-policies/hierarchical-ingress-rules.yaml"
   }

blueprints/factories/README.md

@@ -1,44 +1,79 @@
-# The why and the how of Resource Factories
+# Resource Factories
 
-Terraform modules can be designed - where it makes sense - to implement a resource factory, which is a configuration-driven approach to resource creation meant to:
+This README explains the rationale and high level approach for resource factories, a pattern that is widely used in this repository across modules and in the FAST framework. It also collects pointers to all the different factories implemented in modules to simplify discovery.
 
-- accelerate and rationalize the repetitive creation of common resources, such as firewall rules and subnets
-- enable teams without Terraform specific knowledge to leverage IaC via human-friendly and machine-parseable YAML files
-- make it simple to implement specific requirements and best practices (e.g. "always enable PGA for GCP subnets", or "only allow using regions `europe-west1` and `europe-west3`")
-- codify and centralise business logics and policies (e.g. labels and naming conventions)
-- allow to easily parse and understand sets of specific resources, for documentation purposes
+<!-- BEGIN TOC -->
+- [The why](#the-why)
+- [The how](#the-how)
+- [Factory implementations](#factory-implementations)
+  - [Module-level factory interfaces](#module-level-factory-interfaces)
+  - [Standalone factories](#standalone-factories)
+<!-- END TOC -->
 
-Generally speaking, the configurations for a resource factory consists in one or more YaML files, optionally grouped in folders, that describe resources following a well defined, validable schema, such as in the example below for the subnet factory of the [`net-vpc`](../../modules/net-vpc) module, which allows for the massive creation of subnets for a given VPC.
+## The why
 
-```yaml
-region: europe-west3
-ip_cidr_range: 10.0.0.0/24
-description: Sample Subnet in project project-prod-a, vpc-alpha
-secondary_ip_ranges:
-  secondary-range-a: 192.168.0.0/24
-  secondary-range-b: 192.168.1.0/24
-```
+Managing large sets of uniform resources with Terraform usually involves different teams collaborating on the same codebase, complex authorization processes and checks managed via CI/CD approvals, or even integrating with external systems that manage digital workflows.
 
-Terraform natively supports YaML, JSON and CSV parsing - however Fabric has decided to embrace YaML for the following reasons:
+Factories are a way to simplify all above use cases, by moving repetitive resource definitions out of the Terraform codebase and into sets of files that leverage different formats.
 
-- YaML is easier to parse for a human, and allows for comments and nested, complex structures
-- JSON and CSV can't include comments, which can be used to document configurations, but are often useful to bridge from other systems in automated pipelines
-- JSON is more verbose (reads: longer) and harder to parse visually for humans
-- CSV isn't often expressive enough (e.g. dit doesn't allow for nested structures)
+Using factories, repetive resource creation and management becomes easier
 
-If needed, converting factories to consume JSON is a matter of switching from `yamldecode()` to `jsondecode()` in the right place on each module.
+- for humans who have no direct experience with Terraform, by exposing filesystem hierarchies and YAML-based configuration data
+- for connected systems, by accepting well know data exchange formats like JSON or CSV
+- for external code that needs to enforce checks or policies, by eliminating the need to parse HCL code or Terraform outputs
+- to implement authorization processes or workwflows in CI/CD, by removing the dependency on Terraform and HCL knowledge for the teams involved
 
-## Resource factories in Fabric
+## The how
 
-### Fabric Modules
+Fabric resource-level factories can be broadly split into two different types
 
-- [folder](../../modules/folder/README.md#firewall-policy-factory) and [organization](../../modules/organization/README.md#firewall-policy-factory) implement factories for [hierarchical firewall policies](https://cloud.google.com/vpc/docs/firewall-policies)
-- [net-vpc](../../modules/net-vpc/README.md#subnet-factory) for subnets creation
-- [net-vpc-firewall](../../modules/net-vpc-firewall/README.md#rules-factory) for massive rules creation
+- simple factories that manage one simple resource type (firewalls, VPC-SC policies)
+- complex factories that manage a set of connected resources to implement a complex flow that is usually perceived as a single unit (project creation)
 
-### Dedicated Factories
+The first factory type is implemented at the module level, where one module exposes one or more factories for some of the resources that depend on the main module resource (e.g. firewall rules for a VPC). The main goal with this approach is to simplify resource management at scale by removing the dependency on Terraform and HCL.
 
-- [cloud-identity-group-factory](cloud-identity-group-factory/README.md) for Cloud Identity group
-- [net-vpc-firewall-yaml](net-vpc-firewall-yaml/README.md) for VPC firewall rules across different projects/VPCs
-- [project-factory](project-factory/README.md) for projects
-
+These factories are often designed as module-level interfaces which are then exposed by any module that manages a specific type of resource. All these factories leverage a single `factory_configs` variable, that allows passing in the paths for all the different factories supported in the module.
+
+The second factory type is implemented as a standalone module that internally references other modules, and implements complex management of different resource sets as part of a single process implemented via the factory. The typical example is the project factory, that brings together the project, service accounts, and billing accounts modules to cover all the main aspects of project creation as a single unit.
+
+## Factory implementations
+
+### Module-level factory interfaces
+
+- **BigQuery Analicts Hub rules**
+  - `analytics-hub`
+- **billing budgets**
+  - `billing-account`
+- **Data Catalog tags**
+  - `data-catalog-tag`
+- **Data Catalog tag templates**
+  - `data-catalog-tag-template`
+- **Dataplex Datascan rules**
+  - `dataplex-datascan`
+- **firewall policy rules**
+  - `net-firewall-policy`
+- **hierarchical firewall policies**
+  - `folder`
+  - `project`
+- **IAM custom roles**
+  - `organization`
+  - `project`
+- **organization policies**
+  - `organization`
+  - `folder`
+  - `project`
+- **organization policy custom constraints**
+  - `organization`
+- **DNS response policy rules**
+  - `dns-response-policy`
+- **VPC firewall rules**
+  - `net-vpc-firewall`
+- **VPC subnets**
+  - `net-vpc`
+- **VPC-SC access levels and policies**
+  - `vpc-sc`
+
+### Standalone factories
+
+- **projects**
+  - `project-factory`