Multiple Hub scenario - Bug Report

[mlomat]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Versions

terraform: 1.1.7

azure provider: 2.99.0

Description

Describe the bug

Cannot deploy two hubs in this same region.

Steps to Reproduce

I used your example configuration from Deploy Connectivity with custom settings and I would like to deploy multiple hubs in this same region, because in my use case we have many VPNs in different sites, so each site needs to be separate Hub and app based on requirement will be connected to one or many hubs.

I changed below config to use for both hubs location "westeurope" and it's failing with error.

locals {
  configure_connectivity_resources = {
    settings = {
      hub_networks = [
        {
          enabled = true
          config = {
            address_space                = ["10.100.0.0/16", ]
            location                     = "westeurope"
            link_to_ddos_protection_plan = true
            dns_servers                  = []
            bgp_community                = ""
            subnets                      = []
            virtual_network_gateway = {
              enabled = true
              config = {
                address_prefix           = "10.100.1.0/24"
                gateway_sku_expressroute = "ErGw2AZ"
                gateway_sku_vpn          = ""
              }
            }
            azure_firewall = {
              enabled = true
              config = {
                address_prefix   = "10.100.0.0/24"
                enable_dns_proxy = true
                availability_zones = {
                  zone_1 = true
                  zone_2 = true
                  zone_3 = true
                }
              }
            }
            spoke_virtual_network_resource_ids      = []
            enable_outbound_virtual_network_peering = true
          }
        },
        {
          enabled = true
          config = {
            address_space                = ["10.101.0.0/16", ]
            location                     = "westeurope"
            link_to_ddos_protection_plan = true
            dns_servers                  = []
            bgp_community                = ""
            subnets                      = []
            virtual_network_gateway = {
              enabled = true
              config = {
                address_prefix           = "10.101.1.0/24"
                gateway_sku_expressroute = ""
                gateway_sku_vpn          = "VpnGw2AZ"
              }
            }
            azure_firewall = {
              enabled = false
              config = {
                address_prefix   = ""
                enable_dns_proxy = true
                availability_zones = {
                  zone_1 = true
                  zone_2 = true
                  zone_3 = true
                }
              }
            }
            spoke_virtual_network_resource_ids      = []
            enable_outbound_virtual_network_peering = true
          }
        },
      ]
      vwan_hub_networks = []
      ddos_protection_plan = {
        enabled = true
        config = {
          location = "westeurope"
        }
      }
      dns = {
        enabled = true
        config = {
          location = null
          enable_private_link_by_service = {
            azure_automation_webhook             = true
            azure_automation_dscandhybridworker  = true
            azure_sql_database_sqlserver         = true
            azure_synapse_analytics_sqlserver    = true
            azure_synapse_analytics_sql          = true
            storage_account_blob                 = true
            storage_account_table                = true
            storage_account_queue                = true
       
```     storage_account_file                 = true
            storage_account_web                  = true
            azure_data_lake_file_system_gen2     = true
            azure_cosmos_db_sql                  = true
            azure_cosmos_db_mongodb              = true
            azure_cosmos_db_cassandra            = true
            azure_cosmos_db_gremlin              = true
            azure_cosmos_db_table                = true
            azure_database_for_postgresql_server = true
            azure_database_for_mysql_server      = true
            azure_database_for_mariadb_server    = true
            azure_key_vault                      = true
            azure_kubernetes_service_management  = true
            azure_search_service                 = true
            azure_container_registry             = true
            azure_app_configuration_stores       = true
            azure_backup                         = true
            azure_site_recovery                  = true
            azure_event_hubs_namespace           = true
            azure_service_bus_namespace          = true
            azure_iot_hub                        = true
            azure_relay_namespace                = true
            azure_event_grid_topic               = true
            azure_event_grid_domain              = true
            azure_web_apps_sites                 = true
            azure_machine_learning_workspace     = true
            signalr                              = true
            azure_monitor                        = true
            cognitive_services_account           = true
            azure_file_sync                      = true
            azure_data_factory                   = true
            azure_data_factory_portal            = true
            azure_cache_for_redis                = true
          }
          private_link_locations                                 = []
          public_dns_zones                                       = []
          private_dns_zones                                      = []
          enable_private_dns_zone_virtual_network_link_on_hubs   = true
          enable_private_dns_zone_virtual_network_link_on_spokes = true
        }
      }
    }

    location = "westeurope"
    tags     = null
    advanced = null
  }
}


**Screenshots**
![image](https://user-images.githubusercontent.com/7577121/161997512-d2cbcac8-e168-44d6-a21d-4ac367a70a3d.png)


<!-- If applicable, add screenshots to help explain your problem. -->

[mlomat]

No idea why, but it didn't add screenshot with error...

[krowlandson]

Cannot deploy two hubs in this same region.

Sorry, we should probably document this more clearly. We only support deploying one hub per region unless you use multiple instances of the module with a unique root_id to prevent naming conflicts.

Please can you elaborate a bit more on your use case, as we identified multiple hubs within the same region as a bit of an edge case, so were happy to accept this compromise in the module.

[mlomat]

I hope that this quick diagram will give you some overview. We have multiple sites in the world and very complicated network setup. To simplify it, we want to establish a separate hub for each connection, which will be reused by different projects. It's not a common scenario where you have point-to-point between Azure and On-Prem.

Please let me know if it's clear now.
Best

[krowlandson]

Understood, thank you for the additional diagram @mlomat.

Out of interest, is this a security-based decision to segregate like this, or to handle overlapping addresses, or something else?

To achieve this with the module you will need to define multiple instances of the module and then take either of the following approaches to configuring it:

Unique root_id values

module "connectivity_hub_1" {
  source  = "Azure/caf-enterprise-scale/azurerm"
  version = "1.1.4"

  providers = {
    azurerm              = azurerm
    azurerm.connectivity = azurerm
    azurerm.management   = azurerm
  }

  # Base module configuration settings
  root_parent_id   = "00000000-0000-0000-0000-000000000000"
  root_id          = "hub-1"
  default_location = "westeurope"

  # Disable core landing zones creation
  deploy_core_landing_zones = false

  # Configuration settings for connectivity resources
  deploy_connectivity_resources    = true
  configure_connectivity_resources = {
    # Enter your hub configuration here
  }
  subscription_id_connectivity = "00000000-0000-0000-0000-000000000000"

}

Should result in something like:

Unique resource_prefix values

module "connectivity_hub_2" {
  source  = "Azure/caf-enterprise-scale/azurerm"
  version = "1.1.4"

  providers = {
    azurerm              = azurerm
    azurerm.connectivity = azurerm
    azurerm.management   = azurerm
  }

  # Base module configuration settings
  root_parent_id   = "00000000-0000-0000-0000-000000000000"
  root_id          = "my-root"
  default_location = "westeurope"

  # Disable core landing zones creation
  deploy_core_landing_zones = false

  # Configuration settings for connectivity resources
  deploy_connectivity_resources    = true
  configure_connectivity_resources = {
    # Enter your hub configuration here
    advanced = {
      resource_prefix = "hub-1"
    }
  }
  subscription_id_connectivity = "00000000-0000-0000-0000-000000000000"

}

Should result in something like:

Hope this helps?

[krowlandson]

I'm sure it goes without saying, but you can mix both of the above approaches and repeat X times as needed. However you may wish to consider disabling certain sub-settings.

For example, do you need multiple sets of Private DNS Zones or DDoS Protection Plans?

All of these things are configurable, and you can share certain resources across multiple instances but may require further guidance on this as these are undocumented features within the configure_connectivity_resources.advanced object.

[mlomat]

Thanks, @krowlandson, for replay :)
To answer your first question - design is because of both: segregate and possible handle overlaps in IP addresses.
Regarding DNS Zones in case of IP overlaps we will need to have multiple sets of those. Not sure about DDoS Protection yet. I will test it, but I think that second config should have name module "connectivity_hub_2" instead of "connectivity_hub_1" to match your results ;)

Best
Mateusz

[krowlandson]

I think that second config should have name module "connectivity_hub_2" instead of "connectivity_hub_1" to match your results ;)

True, updated :)

[mlomat]

Thanks @krowlandson - I tested your code and looks good, and I can deploy what I want. I will probably go back in future to ask more about this "configure_connectivity_resources.advanced". For now I was able to deploy two hubs in this same region.