azurerm_mariadb_virtual_network_rule doesn't work cross-subscription

[rmt]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.29

provider "azurerm" { version = "=2.22.0" }

Affected Resource(s)

• azurerm_mariadb_virtual_network_rule

Terraform Configuration Files

resource "azurerm_mariadb_virtual_network_rule" "vnet-rule" {
  count                                = length(var.vnet_subnet_ids)
  name                                 = "${var.server_name}-vnet-rule-${count.index}"
  resource_group_name                  = azurerm_resource_group.default.name
  server_name                          = azurerm_mariadb_server.server.name
  subnet_id                            = var.vnet_subnet_ids[count.index]
}

Debug Output

Current provider's subscription is YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY, and the DB is deployed there.

on modules/mariadb-vnet/main.tf line 61, in resource "azurerm_mariadb_virtual_network_rule" "service_rule":
61: resource "azurerm_mariadb_virtual_network_rule" "service_rule" {
Error: Subnet with ID "/subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/resourceGroups/myResourceGroup/providers/Microsoft.Network/virtualNetworks/myVnetName/subnets/mySubnetName"; was not found: network.SubnetsClient#Get: Failure responding to request: StatusCode=404 -- Original Error: autorest/azure: Service returned an error. Status=404 Code="ResourceGroupNotFound" Message="Resource group 'myResourceGroup' could not be found."

This error only occurs if the subnetId is in a different subscription.

If both the DB and the VNET are in the same subscription, then it works fine.

Using the GUI and the CLI it is possible to add the virtual network rules.

Expected Behavior

The virtual network rule(s) should have been added without any error messages.

When creating a virtual network rule to a subnet in a different subscription, then the resourceGroup should either be looked up in that subscription, or the resourceGroup check should just be skipped, allowing the API call to fail if it doesn't exist. Since subnet_id is a fully fledged ID, it is reasonable to assume that the resourceGroup already exists.

Actual Behavior

The AzureRM provider tried to verify the existence of the subnetIDs resourceGroup in the current subscription, even though the subnetID clearly refers to a different subscription.

Steps to Reproduce

Deploy a DB in subscription A, and try to create a virtual_network_rule to a subnet in subscription B.

Important Factoids

• Public Azure Cloud, both VNET & DB are deployed to the same region

References

[anttisiiskonen]

I'm working with @rmt on this. I believe this problem might be mariadb specific. In mariadb_virtual_network_rule_resource.go there is this code:

        // due to a bug in the API we have to ensure the Subnet's configured correctly or the API call will timeout
	// BUG: https://github.com/Azure/azure-rest-api-specs/issues/3719
	subnetsClient := meta.(*clients.Client).Network.SubnetsClient
	subnetParsedId, err := azure.ParseAzureResourceID(subnetId)
	if err != nil {
		return err
	}

	subnetResourceGroup := subnetParsedId.ResourceGroup
	virtualNetwork := subnetParsedId.Path["virtualNetworks"]
	subnetName := subnetParsedId.Path["subnets"]
	subnet, err := subnetsClient.Get(ctx, subnetResourceGroup, virtualNetwork, subnetName, "")
	if err != nil {
		if utils.ResponseWasNotFound(subnet.Response) {
			return fmt.Errorf("Subnet with ID %q was not found: %+v", subnetId, err)
		}

		return fmt.Errorf("Error obtaining Subnet %q (Virtual Network %q / Resource Group %q: %+v", subnetName, virtualNetwork, subnetResourceGroup, err)
	}

	containsEndpoint := false
	if props := subnet.SubnetPropertiesFormat; props != nil {
		if endpoints := props.ServiceEndpoints; endpoints != nil {
			for _, e := range *endpoints {
				if e.Service == nil {
					continue
				}

				if strings.EqualFold(*e.Service, "Microsoft.Sql") {
					containsEndpoint = true
					break
				}
			}
		}
	}

	if !containsEndpoint {
		return fmt.Errorf("Error creating MariaDb Virtual Network Rule: Subnet %q (Virtual Network %q / Resource Group %q) must contain a Service Endpoint for `Microsoft.Sql`", subnetName, virtualNetwork, subnetResourceGroup)
	}

It seems to be that azurerm mariadb virtual network rule resource is working around a Azure API bug by retrieving the specified subnet by id and verifying that the subnet has a Microsoft.Sql service endpoint, before proceeding on to actually create the rule resource. I believe that this subnet search does not cross subscriptions and therefore the search fails to find the specified vnet.

Azurerm virtual network rule resources for postgres or mysql do not have this code snippet at all -- they do not look at subnet properties before attempting to create the rule. According to the linked github Azure project issue the bug that azurerm is trying to dodge has been fixed already in 2018 for postgres and mysql: Azure/azure-rest-api-specs#3719 (comment)

No word on mariadb though.

[anttisiiskonen]

This explicit subnet check was removed from azurerm for mysql in #5568 and for postgresql in #2056. Could it be simply removed from mariadb as well ..?

[tombuildsstuff]

@anttisiiskonen good spot - probably, but it needs testing for if the API returns a useful error now (it didn't before) - otherwise we can update the SDK call/client to use the specific subscription instead - either way should fix this however 👍

[rmt]

My apologies for the wrong postgresql & mysql mention - I'd only tested those same-subscription. We can remove those labels.

[anttisiiskonen]

@tombuildsstuff I did a simple test setup like this (I had to manually sanitize all outputs, I hope I didn't mess it up):

subscription-A:

• a resource group: 'resource-group-A'
• a virtual network: 'vnet-test-A'
• a subnet with Microsoft.SQL endpoint: 'test-subnet-with-sql-endpoint'
• a subnet without Microsoft.SQL endpoint: 'test-subnet-without-sql-endpoint'

subscription-B:

• a resource group: 'resource-group-B'
• a mariadb instance: 'mariadb-B'

I then created a vnet rule for the database like so:

az mariadb server vnet-rule create  --debug -n test-rule-0 -g resource-group-B -s mariadb-B --subscription subscription-B --subnet /subscriptions/subscription-A/resourceGroups/resource-group-A/providers/Microsoft.Network/virtualNetworks/vnet-test-A/subnets/test-subnet-with-sql-endpoint --ignore-missing-endpoint false

And that succeeds as expected without problems. Rule creation takes maybe 10-30 seconds to complete.

I then attempted to create another vnet rule for the database like so:

az mariadb server vnet-rule create --debug -n test-rule-1 -g resource-group-B -s mariadb-B --subscription subscription-B --subnet /subscriptions/subscription-A/resourceGroups/resource-group-A/providers/Microsoft.Network/virtualNetworks/vnet-test-A/subnets/test-subnet-without-sql-endpoint --ignore-missing-endpoint false

.. and that produces an error like this:

msrest.http_logger : {"error":{"code":"VirtualNetworkRuleBadRequest","message":"Network resource provider returned following error: 'Subnets test-subnet-without-sql-endpoint of virtual network /subscriptions/subscription-A/resourceGroups/resource-group-A/providers/Microsoft.Network/virtualNetworks/vnet-test-A do not have ServiceEndpoints for Microsoft.Sql resources configured. Add Microsoft.Sql to subnet's ServiceEndpoints collection before trying to ACL Microsoft.Sql resources to these subnets.'."}}
msrest.exceptions : Network resource provider returned following error: 'Subnets test-subnet-without-sql-endpoint of virtual network /subscriptions/subscription-A/resourceGroups/resource-group-A/providers/Microsoft.Network/virtualNetworks/vnet-test-A do not have ServiceEndpoints for Microsoft.Sql resources configured. Add Microsoft.Sql to subnet's ServiceEndpoints collection before trying to ACL Microsoft.Sql resources to these subnets.'.
cli.azure.cli.core.util : Network resource provider returned following error: 'Subnets test-subnet-without-sql-endpoint of virtual network /subscriptions/subscription-A/resourceGroups/resource-group-A/providers/Microsoft.Network/virtualNetworks/vnet-test-A do not have ServiceEndpoints for Microsoft.Sql resources configured. Add Microsoft.Sql to subnet's ServiceEndpoints collection before trying to ACL Microsoft.Sql resources to these subnets.'.
Network resource provider returned following error: 'Subnets test-subnet-without-sql-endpoint of virtual network /subscriptions/subscription-A/resourceGroups/resource-group-A/providers/Microsoft.Network/virtualNetworks/vnet-test-A do not have ServiceEndpoints for Microsoft.Sql resources configured. Add Microsoft.Sql to subnet's ServiceEndpoints collection before trying to ACL Microsoft.Sql resources to these subnets.'.

AFAICT the error comes from the Azure API and the response is ~immediate without any delay.

[ghost]

This has been released in version 2.23.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.23.0"
}
# ... other configuration ...

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!