Merge branch 'master' into patch-1

CONTRIBUTING.md

@@ -7,22 +7,22 @@ Contributors are the engine that keeps Fabric alive so if you were or are planni
 - [I just found a bug / have a feature request](#i-just-found-a-bug--have-a-feature-request)
 - [Quick developer workflow](#quick-developer-workflow)
 - [Developer's handbook](#developers-handbook)
-  * [The Zen of Fabric](#the-zen-of-fabric)
-  * [Design principles in action](#design-principles-in-action)
-  * [FAST stage design](#fast-stage-design)
-  * [Style guide reference](#style-guide-reference)
-  * [Interacting with checks and tools](#interacting-with-checks-and-tools)
+  - [The Zen of Fabric](#the-zen-of-fabric)
+  - [Design principles in action](#design-principles-in-action)
+  - [FAST stage design](#fast-stage-design)
+  - [Style guide reference](#style-guide-reference)
+  - [Interacting with checks and tools](#interacting-with-checks-and-tools)
 - [Using and writing tests](#using-and-writing-tests)
-  * [Testing via README.md example blocks.](#testing-via-readmemd-example-blocks)
-    + [Testing examples against an inventory YAML](#testing-examples-against-an-inventory-yaml)
-    + [Using external files](#using-external-files)
-    + [Running tests for specific examples](#running-tests-for-specific-examples)
-    + [Generating the inventory automatically](#generating-the-inventory-automatically)
-    + [Building tests for blueprints](#building-tests-for-blueprints)
-  * [Testing via `tfvars` and `yaml` (aka `tftest`-based tests)](#testing-via-tfvars-and-yaml-aka-tftest-based-tests)
-    + [Generating the inventory for `tftest`-based tests](#generating-the-inventory-for-tftest-based-tests)
-  * [Writing tests in Python (legacy approach)](#writing-tests-in-python-legacy-approach)
-  * [Running tests from a temporary directory](#running-tests-from-a-temporary-directory)
+  - [Testing via README.md example blocks.](#testing-via-readmemd-example-blocks)
+    - [Testing examples against an inventory YAML](#testing-examples-against-an-inventory-yaml)
+    - [Using external files](#using-external-files)
+    - [Running tests for specific examples](#running-tests-for-specific-examples)
+    - [Generating the inventory automatically](#generating-the-inventory-automatically)
+    - [Building tests for blueprints](#building-tests-for-blueprints)
+  - [Testing via `tfvars` and `yaml` (aka `tftest`-based tests)](#testing-via-tfvars-and-yaml-aka-tftest-based-tests)
+    - [Generating the inventory for `tftest`-based tests](#generating-the-inventory-for-tftest-based-tests)
+  - [Writing tests in Python (legacy approach)](#writing-tests-in-python-legacy-approach)
+  - [Running tests from a temporary directory](#running-tests-from-a-temporary-directory)
 - [Fabric tools](#fabric-tools)
 
 ## I just found a bug / have a feature request
@@ -205,11 +205,11 @@ We have several such interfaces defined for IAM, log sinks, organizational polic
 #### Design interfaces to support actual usage
 
 > “When developing a module, look for opportunities to take a little bit of extra suffering upon yourself in order to reduce the suffering of your users.”
-> 
+>
 > “Providing choice is good, but interfaces should be designed to make the common case as simple as possible”
-> 
+>
 > — John Ousterhout in "A Philosophy of Software Design"
- 
+
 Variables should not simply map to the underlying resource attributes, but their **interfaces should be designed to match common use cases** to reduce friction and offer the highest possible degree of legibility.
 
 This translates into different practical approaches:
@@ -300,7 +300,6 @@ module "project" {
 >
 > — John Ousterhout in "A Philosophy of Software Design"
 
-
 Designing variable spaces is one of the most complex aspects to get right, as they are the main entry point through which users consume modules, examples and FAST stages. We always strive to **design small variable spaces by leveraging objects and implementing defaults** so that users can quickly produce highly readable code.
 
 One of many examples of this approach comes from disk support in the `compute-vm` module, where preset defaults allow quick VM management with very few lines of code, and optional variables allow progressively expanding the code when more control is needed.
@@ -688,7 +687,7 @@ In the following sections we describe the three testing approaches we currently
 - [tfvars-based tests](#testing-via-tfvars-and-yaml): allows you to test a module or blueprint by providing variables via tfvar files and an expected plan result in form of an inventory. This type of test is useful, for example, for FAST stages that don't have any examples within their READMEs.
 - [Python-based (legacy) tests](#writing-tests-in-python--legacy-approach-): in some situations you might still want to interact directly with `tftest` via Python, if that's the case, use this method to write custom Python logic to test your module in any way you see fit.
 
-### Testing via README.md example blocks.
+### Testing via README.md example blocks
 
 This is the preferred method to write tests for modules and blueprints. Example-based tests are triggered from [HCL Markdown fenced code blocks](https://docs.github.com/en/get-started/writing-on-github/working-with-advanced-formatting/creating-and-highlighting-code-blocks#syntax-highlighting) in any file named README.md, hence there's no need to create any additional files or revert to Python to write a test. Most of our documentation examples are using this method.
 
@@ -795,6 +794,7 @@ module "private-dns" {
 }
 # tftest modules=1 resources=2 files=records
 ```
+
 ```yaml
 # tftest-file id=records path=records/example.yaml
 A localhost:
@@ -814,6 +814,7 @@ As mentioned before, we use `pytest` as our test runner, so you can use any of t
 Example-based test are named based on the section within the README.md that contains them. You can use this name to select specific tests.
 
 Here we show a few commonly used selection commands:
+
 - Run all examples:
   - `pytest tests/examples/`
 - Run all examples for modules:
@@ -919,7 +920,7 @@ The second approach to testing requires you to:
 - define `yaml` "inventory" files with the plan and output results you want to test
 - declare which of these files need to be run as tests in a `tftest.yaml` file
 
-Let's go through each step in succession, assuming you are testing the new `net-glb` module.
+Let's go through each step in succession, assuming you are testing the new `net-lb-app-ext` module.
 
 First create a new folder under `tests/modules` replacing any dash in the module name with underscores. Note that if you were testing a blueprint the folder would go in `tests/blueprints`.
 
@@ -973,7 +974,7 @@ Create as many pairs of `tfvars`/`yaml` files as you need to test every scenario
 
 ```yaml
 # file: tests/modules/net_glb/tftest.yaml
-module: modules/net-glb
+module: modules/net-lb-app-ext
 # if there are variables shared among all tests you can define a common file
 # common_tfvars:
 #   - defaults.tfvars
@@ -1052,22 +1053,24 @@ You can now use this output to create the inventory file for your test. As menti
 Where possible, we recommend using the testing methods described in the previous sections. However, if you need it, you can still write tests using Python directly.
 
 In general, you should try to use the `plan_summary` fixture, which runs a a terraform plan and returns a `PlanSummary` object. The most important arguments to `plan_summary` are:
+
 - the path of the Terraform module you want to test, relative to the root of the repository
 - a list of paths representing the tfvars file to pass in to terraform. These paths are relative to the python file defining the test.
 
 If successful, `plan_summary` will return a `PlanSummary` object with the `values`, `counts` and `outputs` attributes following the same semantics described in the previous section. You can use this fields to write your custom tests.
 
-Like before let's imagine we're writing a (python) test for `net-glb` module. First create a new folder under `tests/modules` replacing any dash in the module name with underscores. You also need to create an empty `__init__.py` file in it, to ensure `pytest` discovers you new tests automatically.
+Like before let's imagine we're writing a (python) test for `net-lb-app-ext` module. First create a new folder under `tests/modules` replacing any dash in the module name with underscores. You also need to create an empty `__init__.py` file in it, to ensure `pytest` discovers you new tests automatically.
 
 ```bash
 mkdir tests/modules/net_glb
 touch tests/modules/net_glb/__init__.py
 ```
 
 Now create a file containing your tests, e.g. `test_plan.py`:
+
 ```python
 def test_name(plan_summary, tfvars_to_yaml, tmp_path):
-  s = plan_summary('modules/net-glb', tf_var_files=['test-plan.tfvars'])
+  s = plan_summary('modules/net-lb-app-ext', tf_var_files=['test-plan.tfvars'])
   address = 'google_compute_url_map.default'
   assert s.values[address]['project'] == 'my-project'
 ```
@@ -1081,16 +1084,19 @@ Most of the time you can run tests using the `pytest` command as described in pr
 To enable this option, just define the environment variable `TFTEST_COPY` and any tests using the `plan_summary` fixture will automatically run from a temporary directory.
 
 Running tests from temporary directories is useful if:
+
 - you're running tests in parallel using `pytest-xdist`. In this case, just run you tests as follows:
+
   ```bash
   TFTEST_COPY=1 pytest -n 4
   ```
+
 - you're running tests for the `fast/` directory which contain tfvars and auto.tfvars files (which are read by terraform automatically) making your tests fail. In this case, you can run
+
   ```
   TFTEST_COPY=1 pytest fast/
   ```
 
-
 ## Fabric tools
 
 The main tool you will interact with in development is `tfdoc`, used to generate file, output and variable tables in README documents.

README.md

@@ -30,7 +30,7 @@ The current list of modules supports most of the core foundational and networkin
 Currently available modules:
 
 - **foundational** - [billing budget](./modules/billing-budget), [Cloud Identity group](./modules/cloud-identity-group/), [folder](./modules/folder), [service accounts](./modules/iam-service-account), [logging bucket](./modules/logging-bucket), [organization](./modules/organization), [project](./modules/project), [projects-data-source](./modules/projects-data-source)
-- **networking** - [DNS](./modules/dns), [DNS Response Policy](./modules/dns-response-policy/), [Cloud Endpoints](./modules/endpoints), [address reservation](./modules/net-address), [NAT](./modules/net-cloudnat), [VLAN Attachment](./modules/net-vlan-attachment/), [Global Load Balancer (classic)](./modules/net-glb/), [L4 ILB](./modules/net-ilb), [L7 ILB](./modules/net-ilb-l7), [IPSec over Interconnect](./modules/net-ipsec-over-interconnect), [Network LB](./modules/net-nlb), [VPC](./modules/net-vpc), [VPC firewall](./modules/net-vpc-firewall), [VPC firewall policy](./modules/net-vpc-firewall-policy), [VPC peering](./modules/net-vpc-peering), [VPN dynamic](./modules/net-vpn-dynamic), [HA VPN](./modules/net-vpn-ha), [VPN static](./modules/net-vpn-static), [Service Directory](./modules/service-directory)
+- **networking** - [DNS](./modules/dns), [DNS Response Policy](./modules/dns-response-policy/), [Cloud Endpoints](./modules/endpoints), [address reservation](./modules/net-address), [NAT](./modules/net-cloudnat), [VLAN Attachment](./modules/net-vlan-attachment/), [External Application LB](./modules/net-lb-app-ext/), [External Network Passthrough LB](./modules/net-lb-ext), [Internal Application LB](./modules/net-lb-app-int), [Internal Network Passthrough LB](./modules/net-lb-int), [IPSec over Interconnect](./modules/net-ipsec-over-interconnect), [VPC](./modules/net-vpc), [VPC firewall](./modules/net-vpc-firewall), [VPC firewall policy](./modules/net-vpc-firewall-policy), [VPC peering](./modules/net-vpc-peering), [VPN dynamic](./modules/net-vpn-dynamic), [HA VPN](./modules/net-vpn-ha), [VPN static](./modules/net-vpn-static), [Service Directory](./modules/service-directory)
 - **compute** - [VM/VM group](./modules/compute-vm), [MIG](./modules/compute-mig), [COS container](./modules/cloud-config-container/cos-generic-metadata/) (coredns, mysql, onprem, squid), [GKE cluster](./modules/gke-cluster-standard), [GKE hub](./modules/gke-hub), [GKE nodepool](./modules/gke-nodepool)
 - **data** - [AlloyDB instance](./modules/alloydb-instance), [BigQuery dataset](./modules/bigquery-dataset), [Bigtable instance](./modules/bigtable-instance), [Cloud Dataplex](./modules/cloud-dataplex), [Cloud SQL instance](./modules/cloudsql-instance), [Data Catalog Policy Tag](./modules/data-catalog-policy-tag), [Datafusion](./modules/datafusion), [Dataproc](./modules/dataproc), [GCS](./modules/gcs), [Pub/Sub](./modules/pubsub)
 - **development** - [API Gateway](./modules/api-gateway), [Apigee](./modules/apigee), [Artifact Registry](./modules/artifact-registry), [Container Registry](./modules/container-registry), [Cloud Source Repository](./modules/source-repository)

blueprints/apigee/README.md

@@ -20,5 +20,6 @@ The blueprints in this folder contain a variety of deployment scenarios for Apig
 
 The following blueprints demonstrate a set of networking scenarios that can be implemented for Apigee X deployments.
 
-#### Apigee X - Northbound: GLB with PSC Neg, Southbouth: PSC with ILB (L7) and Hybrid NEG
-<a href="./network-patterns/nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg" title="Apigee X - Northbound: GLB with PSC Neg, Southbouth: PSC with ILB (L7) and Hybrid NEG"><img src="./network-patterns/nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg/diagram.png" align="left" width="280px"></a>This [blueprint](./network-patterns/nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg/) shows how to expose an on-prem target backend to clients in the Internet.
+#### Apigee X - Northbound: External Application LB with PSC Neg, Southbouth: PSC with Internal Application LB and Hybrid NEG
+
+<a href="./network-patterns/nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg" title="Northbound: External Application LB with PSC Neg, Southbouth: PSC with Internal Application LB and Hybrid NEG"><img src="./network-patterns/nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg/diagram.png" align="left" width="280px"></a>This [blueprint](./network-patterns/nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg/) shows how to expose an on-prem target backend to clients in the Internet.

blueprints/apigee/bigquery-analytics/main.tf

@@ -88,7 +88,7 @@ module "apigee" {
 }
 
 module "glb" {
-  source              = "../../../modules/net-glb"
+  source              = "../../../modules/net-lb-app-ext"
   name                = "glb"
   project_id          = module.project.project_id
   protocol            = "HTTPS"

blueprints/apigee/network-patterns/README.md

@@ -2,5 +2,6 @@
 
 The blueprints in this folder demonstrate a set of networking scenarios that can be implemented for Apigee X deployments.
 
-## Apigee X - Northbound: GLB with PSC Neg, Southbouth: PSC with ILB (L7) and Hybrid NEG
-<a href="./nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg" title="Apigee X - Northbound: GLB with PSC Neg, Southbouth: PSC with ILB (L7) and Hybrid NEG"><img src="./nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg/diagram.png" align="left" width="280px"></a>This [blueprint](./nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg/) shows how to expose an on-prem target backend to clients in the Internet.g
+## Northbound: External Application LB with PSC Neg, Southbouth: PSC with Internal Application LB and Hybrid NEG
+
+<a href="./nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg" title="Northbound: External Application LB with PSC Neg, Southbouth: PSC with Internal Application LB and Hybrid NEG"><img src="./nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg/diagram.png" align="left" width="280px"></a>This [blueprint](./nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg/) shows how to expose an on-prem target backend to clients in the Internet.g

blueprints/apigee/network-patterns/nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg/README.md

@@ -1,4 +1,4 @@
-# Apigee X - Northbound GLB with PSC Neg, Southbouth PSC with ILB (L7) and Hybrid NEG
+# Apigee X - Northbound: External Application LB with PSC Neg, Southbouth: PSC with Internal Application LB and Hybrid NEG
 
 The following blueprint shows how to expose an on-prem target backend to clients in the Internet.
 

blueprints/apigee/network-patterns/nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg/apigee_nb.tf

@@ -15,7 +15,7 @@
  */
 
 module "glb" {
-  source              = "../../../../modules/net-glb"
+  source              = "../../../../modules/net-lb-app-ext"
   name                = "glb"
   project_id          = module.apigee_project.project_id
   protocol            = "HTTPS"

blueprints/apigee/network-patterns/nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg/apigee_sb.tf

@@ -15,7 +15,7 @@
  */
 
 module "apigee_ilb_l7" {
-  source     = "../../../../modules/net-ilb-l7"
+  source     = "../../../../modules/net-lb-app-int"
   name       = "apigee-ilb"
   project_id = module.apigee_project.project_id
   region     = var.region

blueprints/apigee/network-patterns/nb-glb-psc-neg-sb-psc-ilbl7-hybrid-neg/onprem.tf

@@ -116,7 +116,7 @@ module "mig" {
 }
 
 module "onprem_ilb_l7" {
-  source     = "../../../../modules/net-ilb-l7"
+  source     = "../../../../modules/net-lb-app-int"
   name       = "ilb"
   project_id = module.onprem_project.project_id
   region     = var.region

blueprints/cloud-operations/adfs/main.tf

@@ -96,7 +96,7 @@ module "server" {
 }
 
 module "glb" {
-  source     = "../../../modules/net-glb"
+  source     = "../../../modules/net-lb-app-ext"
   name       = "${var.prefix}-glb"
   project_id = module.project.project_id
   protocol   = "HTTPS"

blueprints/cloud-operations/network-dashboard/README.md

@@ -72,7 +72,7 @@ Refer to the [Cloud Function deployment instructions](./deploy-cloud-function/)
 ## Assumptions and limitations
 
 - The tool assumes all VPCs in peering groups are within the same organization, except for PSA peerings.
-- The tool will only fetch subnet utilization data from the PSA peerings (not the VMs, ILB or routes usage).
+- The tool will only fetch subnet utilization data from the PSA peerings (not the VMs, LB or routes usage).
 - The tool assumes global routing is ON, this impacts dynamic routes usage calculation.
 - The tool assumes custom routes importing/exporting is ON, this impacts static and dynamic routes usage calculation.
 - The tool assumes all networks in peering groups have the same global routing and custom routes sharing configuration.

blueprints/data-solutions/data-platform-minimal/01-landing.tf

@@ -16,7 +16,7 @@
 
 locals {
   iam_lnd = {
-    "roles/storage.objectCreator" = [module.land-sa-cs-0.iam_email]
+    "roles/storage.objectCreator" = [module.land-sa-0.iam_email]
     "roles/storage.objectViewer"  = [module.processing-sa-cmp-0.iam_email]
     "roles/storage.objectAdmin"   = [module.processing-sa-0.iam_email]
   }
@@ -36,6 +36,9 @@ module "land-project" {
   iam          = var.project_config.billing_account_id != null ? local.iam_lnd : null
   iam_additive = var.project_config.billing_account_id == null ? local.iam_lnd : null
   services = [
+    "bigquery.googleapis.com",
+    "bigqueryreservation.googleapis.com",
+    "bigquerystorage.googleapis.com",
     "cloudkms.googleapis.com",
     "cloudresourcemanager.googleapis.com",
     "iam.googleapis.com",
@@ -52,12 +55,12 @@ module "land-project" {
 
 # Cloud Storage
 
-module "land-sa-cs-0" {
+module "land-sa-0" {
   source       = "../../../modules/iam-service-account"
   project_id   = module.land-project.project_id
   prefix       = var.prefix
-  name         = "lnd-cs-0"
-  display_name = "Data platform GCS landing service account."
+  name         = "lnd-sa-0"
+  display_name = "Data platform landing zone service account."
   iam = {
     "roles/iam.serviceAccountTokenCreator" = [
       local.groups_iam.data-engineers
@@ -75,3 +78,11 @@ module "land-cs-0" {
   encryption_key = var.service_encryption_keys.storage
   force_destroy  = var.data_force_destroy
 }
+
+module "land-bq-0" {
+  source         = "../../../modules/bigquery-dataset"
+  project_id     = module.land-project.project_id
+  id             = "${replace(var.prefix, "-", "_")}_lnd_bq_0"
+  location       = var.location
+  encryption_key = var.service_encryption_keys.bq
+}

blueprints/data-solutions/data-platform-minimal/02-processing.tf

@@ -49,16 +49,14 @@ locals {
   }
   processing_subnet = (
     local.use_shared_vpc
-    ? var.network_config.subnet_self_links.processing_transformation
+    ? var.network_config.subnet_self_link
     : module.processing-vpc.0.subnet_self_links["${var.region}/${var.prefix}-processing"]
   )
   processing_vpc = (
     local.use_shared_vpc
     ? var.network_config.network_self_link
     : module.processing-vpc.0.self_link
   )
-
-
 }
 
 module "processing-project" {