Blueprint: XGLB hybrid NEG internal

blueprints/networking/glb-hybrid-neg-internal/README.md

@@ -0,0 +1,82 @@
+# XGLB and multi-reginoal daisy-chaining through hybrid NEGs
+
+The blueprint shows the experimental use of hybrid NEGs behind eXternal Global Load Balancers (XGLBs) to connect to GCP instances living in spoke VPCs and behind Network Virtual Appliances (NVAs).
+
+<p align="center"> <img src="diagram.png" width="700"> </p>
+
+This allows users to not configure per-destination-VM NAT rules in the NVAs.
+
+The user traffic will enter the XGLB, it will go across the NVAs and it will be routed to the destination VMs (or the ILBs behind the VMs) in the spokes.
+
+## What the blueprint creates
+
+This is what the blueprint brings up, using the default module values.
+The ids `r1` and `r2` are used to identify two regions. By default, `europe-west1` and `europe-west2`.
+
+- Projects: landing, spoke-01
+
+- VPCs and subnets
+	+ landing-untrusted: r1 - 192.168.1.0/24 and r2 - 192.168.2.0/24
+	+ landing-trusted: r1 - 192.168.11.0/24 and r2 - 192.168.22.0/24
+	+ spoke-01: r1 - 192.168.101.0/24 and r2 - 192.168.102.0/24
+
+- Cloud NAT
+	+ landing-untrusted (both for r1 and r2)
+	+ in spoke-01 (both for r1 and r2) - this is just for test purposes, so you VMs can automatically install nginx, even if NVAs are still not ready
+
+- VMs
+	+ NVAs in MIGs in the landing project, both in r1 and r2, with NICs in the untrusted and in the trusted VPCs
+	+ Test VMs, in spoke-01, both in r1 and r2. Optionally, deployed in MIGs
+
+- Hybrid NEGs in the untrusted VPC, both in r1 and r2, either pointing to the test VMs in the spoke or -optionally- to ILBs in the spokes (if test VMs are deployed as MIGs)
+
+- Internal Load balancers (L4 ILBs)
+	+ in the untrusted VPC, pointing to NVA MIGs, both in r1 and r2. Their VIPs are used by custom routes in the untrusted VPC, so that all traffic that arrives in the untrusted VPC destined for the test VMs in the spoke is sent through the NVAs
+	+ optionally, in the spokes. They are created if the user decides to deploy the test VMs as MIGs
+
+- External Global Load balancer (XGLB) in the untrusted VPC, using the hybrid NEGs as its backends
+
+## Health Checks
+
+Google Cloud sends [health checks](https://cloud.google.com/load-balancing/docs/health-checks) using [specific IP ranges](https://cloud.google.com/load-balancing/docs/health-checks#fw-netlb). Each VPC uses [implicit routes](https://cloud.google.com/vpc/docs/routes#special_return_paths) to send the health check replies back to Google.
+
+At the moment of writing, when Google Cloud sends out [health checks](https://cloud.google.com/load-balancing/docs/health-checks) against backend services, it expects replies to come back from the same VPC where they have been sent out to.
+
+Given the XGLB lives in the untrusted VPC, its backend service health checks are sent out to that VPC, and so the replies are expected from it. Anyway, the destinations of the health checks are the test VMs in the spokes.
+
+The blueprint configures some custom routes in the untrusted VPC and routing/NAT rules in the NVAs, so that health checks reach the test VMs through the NVAs, and replies come back through the NVAs in the untrusted VPC. Without these configurations health checks will fail and backend services won't be reachable.
+
+Specifically:
+
+- we create two custom routes in the untrusted VPC (one per region) so that traffic for the spoke subnets is sent to the VIP of the L4 ILBs in front of the NVAs
+
+- we configure the NVAs so they know how to route traffic to the spokes via the trusted VPC gateway
+
+- we configure the NVAs to s-NAT (specifically, masquerade) health checks traffic destined to the test VMs
+
+## Change the test_vms_behind_ilb variable
+
+Through the `test_vms_behind_ilb` variable you can decide whether test VMs in the spoke will be deployed as MIGs with ILBs in front. This will also configure NEGs, so they point to the ILB VIPs, instead of the VM IPs.
+
+At the moment, every time a user changes the configuration of a NEG, the NEG is recreated. When this happens, the provider doesn't check if it is used by other resources, such as XGLB backend services. Until this doesn't get fixed, every time you'll need to change the NEG configuration (i.e. when changing the variable `test_vms_behind_ilb`) you'll have to workaround it. Here is how:
+
+- Destroy the existing backend service: `terraform destroy -target 'module.hybrid-glb.google_compute_backend_service.default["default"]'`
+
+- Change the variable `test_vms_behind_ilb`
+
+- run `terraform apply`
+<!-- BEGIN TFDOC -->
+
+## Variables
+
+| name | description | type | required | default |
+|---|---|:---:|:---:|:---:|
+| [prefix](variables.tf#L17) | Prefix used for resource names. | <code>string</code> | ✓ |  |
+| [projects_create](variables.tf#L26) | Parameters for the creation of the new project. | <code title="object&#40;&#123;&#10;  billing_account_id &#61; string&#10;  parent             &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code>null</code> |
+| [region_configs](variables.tf#L35) | The primary and secondary region parameters. | <code title="object&#40;&#123;&#10;  r1 &#61; object&#40;&#123;&#10;    region_name &#61; string&#10;    zone        &#61; string&#10;  &#125;&#41;&#10;  r2 &#61; object&#40;&#123;&#10;    region_name &#61; string&#10;    zone        &#61; string&#10;  &#125;&#41;&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code title="&#123;&#10;  r1 &#61; &#123;&#10;    region_name &#61; &#34;europe-west1&#34;&#10;    zone        &#61; &#34;europe-west1-b&#34;&#10;  &#125;&#10;  r2 &#61; &#123;&#10;    region_name &#61; &#34;europe-west2&#34;&#10;    zone        &#61; &#34;europe-west2-b&#34;&#10;  &#125;&#10;&#125;">&#123;&#8230;&#125;</code> |
+| [test_vms_behind_ilb](variables.tf#L59) | Whether there should be an ILB L4 in front of the test VMs in the spoke. | <code>string</code> |  | <code>&#34;false&#34;</code> |
+| [vpc_landing_trusted_config](variables.tf#L77) | The configuration of the landing trusted VPC | <code title="object&#40;&#123;&#10;  r1_cidr &#61; string&#10;  r2_cidr &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code title="&#123;&#10;  r1_cidr &#61; &#34;192.168.11.0&#47;24&#34;,&#10;  r2_cidr &#61; &#34;192.168.22.0&#47;24&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
+| [vpc_landing_untrusted_config](variables.tf#L65) | The configuration of the landing untrusted VPC | <code title="object&#40;&#123;&#10;  r1_cidr &#61; string&#10;  r2_cidr &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code title="&#123;&#10;  r1_cidr &#61; &#34;192.168.1.0&#47;24&#34;,&#10;  r2_cidr &#61; &#34;192.168.2.0&#47;24&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
+| [vpc_spoke_config](variables.tf#L89) | The configuration of the spoke-01 VPC | <code title="object&#40;&#123;&#10;  r1_cidr &#61; string&#10;  r2_cidr &#61; string&#10;&#125;&#41;">object&#40;&#123;&#8230;&#125;&#41;</code> |  | <code title="&#123;&#10;  r1_cidr &#61; &#34;192.168.101.0&#47;24&#34;,&#10;  r2_cidr &#61; &#34;192.168.102.0&#47;24&#34;&#10;&#125;">&#123;&#8230;&#125;</code> |
+
+<!-- END TFDOC -->

blueprints/networking/glb-hybrid-neg-internal/data/nva-startup-script.tftpl

@@ -0,0 +1,42 @@
+#!/bin/bash
+
+# Copyright 2023 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+echo 'Enabling IP forwarding'
+sed '/net.ipv4.ip_forward=1/s/^#//g' -i /etc/sysctl.conf &&
+sysctl -p /etc/sysctl.conf &&
+/etc/init.d/procps restart
+
+echo 'Setting routes'
+ip route add ${spoke-r1-subnet} via ${gateway-trusted} dev ens5
+ip route add ${spoke-r2-subnet} via ${gateway-trusted} dev ens5
+
+echo 'Setting NAT masquerade, so that HCs can reach the spoke through the NVA using the trusted intf source IP'
+iptables \
+  -t nat \
+  -A POSTROUTING \
+  -s 130.211.0.0/22,35.191.0.0/16 \
+  -d ${spoke-r1-subnet} \
+  -p tcp \
+  --dport 80 \
+  -j MASQUERADE
+iptables \
+  -t nat \
+  -A POSTROUTING \
+  -s 130.211.0.0/22,35.191.0.0/16 \
+  -d ${spoke-r2-subnet} \
+  -p tcp \
+  --dport 80 \
+  -j MASQUERADE

blueprints/networking/glb-hybrid-neg-internal/landing-hub.tf

@@ -0,0 +1,247 @@
+/**
+ * Copyright 2023 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+################################################################################
+#                                Base Hierarchy                                #
+################################################################################
+
+module "project_landing" {
+  source = "git::https://github.com/terraform-google-modules/cloud-foundation-fabric//modules/project"
+  billing_account = (var.projects_create != null
+    ? var.projects_create.billing_account_id
+    : null
+  )
+  name = "landing"
+  parent = (var.projects_create != null
+    ? var.projects_create.parent
+    : null
+  )
+  prefix         = var.prefix
+  project_create = var.projects_create != null
+
+  services = [
+    "compute.googleapis.com",
+    "networkmanagement.googleapis.com",
+    # Logging and Monitoring
+    "logging.googleapis.com",
+    "monitoring.googleapis.com"
+  ]
+}
+
+################################################################################
+#                                  Networking                                  #
+################################################################################
+
+module "vpc_landing_untrusted" {
+  source     = "git::https://github.com/terraform-google-modules/cloud-foundation-fabric//modules/net-vpc"
+  project_id = module.project_landing.project_id
+  name       = "landing-untrusted"
+
+  routes = {
+    spoke1-r1 = {
+      dest_range    = var.vpc_spoke_config.r1_cidr
+      next_hop_type = "ilb"
+      next_hop      = module.nva_untrusted_ilbs["r1"].forwarding_rule_self_link
+    }
+    spoke1-r2 = {
+      dest_range    = var.vpc_spoke_config.r2_cidr
+      next_hop_type = "ilb"
+      next_hop      = module.nva_untrusted_ilbs["r2"].forwarding_rule_self_link
+    }
+  }
+
+  subnets = [
+    {
+      ip_cidr_range = var.vpc_landing_untrusted_config.r1_cidr
+      name          = "untrusted-${var.region_configs.r1.region_name}"
+      region        = var.region_configs.r1.region_name
+    },
+    {
+      ip_cidr_range = var.vpc_landing_untrusted_config.r2_cidr
+      name          = "untrusted-${var.region_configs.r2.region_name}"
+      region        = var.region_configs.r2.region_name
+    }
+  ]
+}
+
+module "vpc_landing_trusted" {
+  source     = "git::https://github.com/terraform-google-modules/cloud-foundation-fabric//modules/net-vpc"
+  project_id = module.project_landing.project_id
+  name       = "landing-trusted"
+  subnets = [
+    {
+      ip_cidr_range = var.vpc_landing_trusted_config.r1_cidr
+      name          = "trusted-${var.region_configs.r1.region_name}"
+      region        = var.region_configs.r1.region_name
+    },
+    {
+      ip_cidr_range = var.vpc_landing_trusted_config.r2_cidr
+      name          = "trusted-${var.region_configs.r2.region_name}"
+      region        = var.region_configs.r2.region_name
+    }
+  ]
+}
+
+module "firewall_landing_untrusted" {
+  source     = "git::https://github.com/terraform-google-modules/cloud-foundation-fabric//modules/net-vpc-firewall"
+  project_id = module.project_landing.project_id
+  network    = module.vpc_landing_untrusted.name
+
+  ingress_rules = {
+    allow-ssh-from-hcs = {
+      description = "Allow health checks to NVAs coming on port 22."
+      targets     = ["ssh"]
+      source_ranges = [
+        "130.211.0.0/22",
+        "35.191.0.0/16"
+      ]
+      rules = [{ protocol = "tcp", ports = [22] }]
+    }
+  }
+}
+
+module "nats_landing" {
+  for_each       = var.region_configs
+  source         = "git::https://github.com/terraform-google-modules/cloud-foundation-fabric//modules/net-cloudnat"
+  project_id     = module.project_landing.project_id
+  region         = each.value.region_name
+  name           = "nat-${each.value.region_name}"
+  router_network = module.vpc_landing_untrusted.self_link
+}
+
+module "nva_instance_templates" {
+  for_each               = var.region_configs
+  source                 = "git::https://github.com/terraform-google-modules/cloud-foundation-fabric//modules/compute-vm"
+  project_id             = module.project_landing.project_id
+  can_ip_forward         = true
+  create_template        = true
+  name                   = "nva-${each.value.region_name}"
+  service_account_create = true
+  zone                   = each.value.zone
+
+  metadata = {
+    startup-script = templatefile(
+      "${path.module}/data/nva-startup-script.tftpl",
+      {
+        gateway-trusted = cidrhost(module.vpc_landing_trusted.subnet_ips["${each.value.region_name}/trusted-${each.value.region_name}"], 1)
+        spoke-r1-subnet = module.vpc_spoke_01.subnet_ips["${var.region_configs.r1.region_name}/spoke-01-${var.region_configs.r1.region_name}"]
+        spoke-r2-subnet = module.vpc_spoke_01.subnet_ips["${var.region_configs.r2.region_name}/spoke-01-${var.region_configs.r2.region_name}"]
+      }
+    )
+  }
+
+  network_interfaces = [
+    {
+      network    = module.vpc_landing_untrusted.self_link
+      subnetwork = module.vpc_landing_untrusted.subnet_self_links["${each.value.region_name}/untrusted-${each.value.region_name}"]
+    },
+    {
+      network    = module.vpc_landing_trusted.self_link
+      subnetwork = module.vpc_landing_trusted.subnet_self_links["${each.value.region_name}/trusted-${each.value.region_name}"]
+    }
+  ]
+
+  tags = [
+    "http-server",
+    "https-server",
+    "ssh"
+  ]
+}
+
+module "nva_migs" {
+  for_each          = var.region_configs
+  source            = "git::https://github.com/terraform-google-modules/cloud-foundation-fabric//modules/compute-mig"
+  project_id        = module.project_landing.project_id
+  location          = each.value.zone
+  name              = "nva-${each.value.region_name}"
+  target_size       = 1
+  instance_template = module.nva_instance_templates[each.key].template.self_link
+}
+
+module "nva_untrusted_ilbs" {
+  for_each      = var.region_configs
+  source        = "git::https://github.com/terraform-google-modules/cloud-foundation-fabric//modules/net-ilb"
+  project_id    = module.project_landing.project_id
+  region        = each.value.region_name
+  name          = "nva-ilb-${each.value.region_name}"
+  service_label = "nva-ilb-${each.value.region_name}"
+  vpc_config = {
+    network    = module.vpc_landing_untrusted.self_link
+    subnetwork = module.vpc_landing_untrusted.subnet_self_links["${each.value.region_name}/untrusted-${each.value.region_name}"]
+  }
+  backends = [{
+    group = module.nva_migs[each.key].group_manager.instance_group
+  }]
+  health_check_config = {
+    tcp = {
+      port = 22
+    }
+  }
+}
+
+module "hybrid-glb" {
+  source     = "git::https://github.com/terraform-google-modules/cloud-foundation-fabric//modules/net-glb"
+  project_id = module.project_landing.project_id
+  name       = "hybrid-glb"
+  backend_service_configs = {
+    default = {
+      backends = [
+        {
+          backend        = "neg-r1"
+          balancing_mode = "RATE"
+          max_rate       = { per_endpoint = 100 }
+        },
+        {
+          backend        = "neg-r2"
+          balancing_mode = "RATE"
+          max_rate       = { per_endpoint = 100 }
+        }
+      ]
+    }
+  }
+  neg_configs = {
+    neg-r1 = {
+      hybrid = {
+        network = module.vpc_landing_untrusted.name
+        zone    = var.region_configs.r1.zone
+        endpoints = {
+          r1 = {
+            ip_address = (var.test_vms_behind_ilb
+              ? module.test_vm_ilbs["r1"].forwarding_rule_address
+              : module.test_vms["r1"].internal_ip
+            )
+            port = 80
+          }
+        }
+      }
+    }
+    neg-r2 = {
+      hybrid = {
+        network = module.vpc_landing_untrusted.name
+        zone    = var.region_configs.r2.zone
+        endpoints = {
+          r2 = {
+            ip_address = (var.test_vms_behind_ilb
+              ? module.test_vm_ilbs["r2"].forwarding_rule_address
+              : module.test_vms["r2"].internal_ip
+            )
+            port = 80
+          }
+        }
+      }
+    }
+  }
+}

blueprints/networking/glb-hybrid-neg-internal/outputs.tf

@@ -0,0 +1,15 @@
+/**
+ * Copyright 2023 Google LLC
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */