Azure Firewall: Updating multiple IP Group objects at the same time causes firewall/policy to fail

[sponte]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

1.3.4

AzureRM Provider Version

2.99, 3.37.0

Affected Resource(s)/Data Source(s)

azurerm_ip_group

Terraform Configuration Files

# Start with this

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test" {
  name     = "acctestRG-network-%d"
  location = "%s"
}

resource "azurerm_ip_group" "test1" {
  name                = "acceptanceTestIpGroup1"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name

  cidrs = ["172.16.240.0/20"]

  tags = {
    environment = "Production"
    cost_center = "MSFT"
  }
}

resource "azurerm_ip_group" "test2" {
  name                = "acceptanceTestIpGroup2"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name

  cidrs = ["172.17.240.0/20"]

  tags = {
    environment = "Production"
    cost_center = "MSFT"
  }
}

resource "azurerm_firewall_policy" "test" {
  name                = "fwpol-test-policy"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
}

resource "azurerm_firewall_policy_rule_collection_group" "test" {
  name               = "fwpol-test"
  firewall_policy_id = azurerm_firewall_policy.test.id
  priority           = 100

  network_rule_collection {
    name     = "network-rule-collection1"
    priority = 100
    action   = "Allow"
    rule {
      name                  = "network-rule-collection1-rule1"
      protocols             = ["TCP"]
      source_ip_groups      = [azurerm_ip_group.test1.id]
      destination_ip_groups = [azurerm_ip_group.test2.id]
      destination_ports     = ["443"]
    }
  }

  network_rule_collection {
    name     = "network-rule-collection2"
    priority = 200
    action   = "Allow"
    rule {
      name                  = "network-rule-collection1-rule1"
      protocols             = ["TCP"]
      source_ip_groups      = [azurerm_ip_group.test2.id]
      destination_ip_groups = [azurerm_ip_group.test1.id]
      destination_ports     = ["443"]
    }
  }
}

resource "azurerm_virtual_network" "test" {
  name                = "testvnet"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
}

resource "azurerm_subnet" "test" {
  name                 = "AzureFirewallSubnet"
  resource_group_name  = azurerm_resource_group.test.name
  virtual_network_name = azurerm_virtual_network.test.name
  address_prefixes     = ["10.0.1.0/24"]
}

resource "azurerm_public_ip" "test" {
  name                = "pip-fw"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
  allocation_method   = "Static"
  sku                 = "Standard"
}

resource "azurerm_firewall" "test" {
  name                = "testfirewall"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
  sku_name            = "AZFW_VNet"
  sku_tier            = "Standard"

  firewall_policy_id  = azurerm_firewall_policy.test.id

  ip_configuration {
    name                 = "configuration"
    subnet_id            = azurerm_subnet.test.id
    public_ip_address_id = azurerm_public_ip.test.id
  }
}





# Update to that

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "test" {
  name     = "acctestRG-network-%d"
  location = "%s"
}

resource "azurerm_ip_group" "test1" {
  name                = "acceptanceTestIpGroup1"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name

  cidrs = ["172.16.240.0/20", "172.18.240.0/20"]

  tags = {
    environment = "Production"
    cost_center = "MSFT"
  }
}

resource "azurerm_ip_group" "test2" {
  name                = "acceptanceTestIpGroup2"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name

  cidrs = ["172.17.240.0/20", "172.19.240.0/20"]

  tags = {
    environment = "Production"
    cost_center = "MSFT"
  }
}

resource "azurerm_firewall_policy" "test" {
  name                = "fwpol-test-policy"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
}

resource "azurerm_firewall_policy_rule_collection_group" "test" {
  name               = "fwpol-test"
  firewall_policy_id = azurerm_firewall_policy.test.id
  priority           = 100

  network_rule_collection {
    name     = "network-rule-collection1"
    priority = 100
    action   = "Allow"
    rule {
      name                  = "network-rule-collection1-rule1"
      protocols             = ["TCP"]
      source_ip_groups      = [azurerm_ip_group.test1.id]
      destination_ip_groups = [azurerm_ip_group.test2.id]
      destination_ports     = ["443"]
    }
  }

  network_rule_collection {
    name     = "network-rule-collection2"
    priority = 200
    action   = "Allow"
    rule {
      name                  = "network-rule-collection1-rule1"
      protocols             = ["TCP"]
      source_ip_groups      = [azurerm_ip_group.test2.id]
      destination_ip_groups = [azurerm_ip_group.test1.id]
      destination_ports     = ["443"]
    }
  }
}

resource "azurerm_virtual_network" "test" {
  name                = "testvnet"
  address_space       = ["10.0.0.0/16"]
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
}

resource "azurerm_subnet" "test" {
  name                 = "AzureFirewallSubnet"
  resource_group_name  = azurerm_resource_group.test.name
  virtual_network_name = azurerm_virtual_network.test.name
  address_prefixes     = ["10.0.1.0/24"]
}

resource "azurerm_public_ip" "test" {
  name                = "pip-fw"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
  allocation_method   = "Static"
  sku                 = "Standard"
}

resource "azurerm_firewall" "test" {
  name                = "testfirewall"
  location            = azurerm_resource_group.test.location
  resource_group_name = azurerm_resource_group.test.name
  sku_name            = "AZFW_VNet"
  sku_tier            = "Standard"

  firewall_policy_id  = azurerm_firewall_policy.test.id

  ip_configuration {
    name                 = "configuration"
    subnet_id            = azurerm_subnet.test.id
    public_ip_address_id = azurerm_public_ip.test.id
  }
}

Debug Output/Panic Output

testcase.go:110: Step 3/4 error: Error running apply: exit status 1
        
        Error: waiting for the completion of Ip Group: (Name "acceptanceTestIpGroup1" / Resource Group "acctestRG-network-230103161534034082"): Code="IpGroupsUpdateFailed" Message="Put on IP Groups acceptanceTestIpGroup1 Failed with 1 faulted referenced firewalls"
        
          with azurerm_ip_group.test1,
          on terraform_plugin_test.tf line 11, in resource "azurerm_ip_group" "test1":
          11: resource "azurerm_ip_group" "test1" {
        
    testing_new.go:84: Error running post-test destroy, there may be dangling resources: exit status 1
        
        Error: waiting for deleting "fwpol-test" (Resource Group "acctestRG-network-230103161534034082" / Policy: "fwpol-test-policy"): Code="FirewallPolicyUpdateFailed" Message="Put on Firewall Policy fwpol-test-policy Failed with 1 faulted referenced firewalls"

Expected Behaviour

IP Groups that are connected with Azure Firewall or Azure Firewall Policy resources should only update one group at a time

Actual Behaviour

Multiple updates happen at the same time causing Azure Firewall to go into faulted state

Steps to Reproduce

Run 1st example from the above configuration example, then update the config of both azurerm_ip_group objects and run apply again

Important Factoids

No response

References

From https://learn.microsoft.com/en-us/azure/firewall/overview

• Configuration updates may take five minutes on average
• An Azure Firewall configuration update can take three to five minutes on average, and parallel updates aren't supported.
• A fix is being investigated.

[github-actions]

This functionality has been released in v3.40.0 of the Terraform Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.