Azurerm_frontdoor with v2.24.0 breaks when azure frontdoor is edited in portal.

[andrstor]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.21
+ provider.azurerm v2.24.0

Affected Resource(s)

• azurerm_frontdoor

Terraform Configuration Files

provider "azurerm" {
  version = "=2.24.0"
  features {} # https://www.terraform.io/docs/providers/azurerm/index.html#features
}

resource "azurerm_resource_group" "example" {
  name     = "andreastester"
  location = "norway east"
}

resource "azurerm_frontdoor" "example" {
  name                                         = "andreastester"
  resource_group_name                          = azurerm_resource_group.example.name
  enforce_backend_pools_certificate_name_check = false

  routing_rule {
    name               = "exampleRoutingRule1"
    accepted_protocols = ["Http", "Https"]
    patterns_to_match  = ["/*"]
    frontend_endpoints = ["exampleFrontendEndpoint1"]
    forwarding_configuration {
      forwarding_protocol = "MatchRequest"
      backend_pool_name   = "exampleBackendBing"
    }
  }

  backend_pool_load_balancing {
    name = "exampleLoadBalancingSettings1"
  }

  backend_pool_health_probe {
    name = "exampleHealthProbeSetting1"
    
  }

  backend_pool {
    name = "exampleBackendBing"
    backend {
      host_header = "www.bing.com"
      address     = "www.bing.com"
      http_port   = 80
      https_port  = 443
    }

    load_balancing_name = "exampleLoadBalancingSettings1"
    health_probe_name   = "exampleHealthProbeSetting1"
  }

  frontend_endpoint {
    name                              = "exampleFrontendEndpoint1"
    host_name                         = "andreastester.azurefd.net"
    custom_https_provisioning_enabled = false
  }
}

Debug Output

https://gist.github.com/andrstor/0aa07440e0a01befb23351db3257340f

Panic Output

Expected Behavior

Terraform identifies that no changes are required or tries to recover its state.

Actual Behavior

Error: flattening backend_pool: ID was missing the healthProbeSettings element

Steps to Reproduce

1. terraform apply
2. Do anything in the azure portal that trigges a change. For instance add a rule engine rule to the routing rule.
3. terraform plan

You can also undo the manual change again, the resource is still broken for azurerm v2.24.0. This works with v2.23.0

Important Factoids

None

References

• Nomalised Azure frontdoor IDs #8046

[eliasgrueninger]

Same issue here but on another level:
Error: flattening frontend_endpoint: ID was missing the frontDoorWebApplicationFirewallPolicies element

[lyubomirr]

Same behaviour when trying to import the resource.

[MelHarbour]

Seeing the same issue as @eliasgrueninger on the Firewall Policies element, but we're running 2.21, which implies that it's not necessarily a change in the most recent version of the provider, but could be a change at the Azure end?

[nmiodice]

I found a similar issue (#8231) and did not realize there was an open issue already.

The only difference in my case is that the AFD & related resources were not edited in the Azure Portal - these resources are fully managed by Terraform. cc @tombuildsstuff , that is the only difference I see between these two issues, though the root cause is likely the same.

[kevinchabreck]

I experienced this issue when trying to update the resource, and then again when trying to import it after removing it from the terraform state.

import azurerm_frontdoor.fd <my frontdoor resource ID>
azurerm_frontdoor.fd: Importing from ID "<my frontdoor resource ID>"...
azurerm_frontdoor.fd: Import prepared!
  Prepared azurerm_frontdoor for import
azurerm_frontdoor.fd: Refreshing state... [id=<my frontdoor resource ID>]

Error: flattening `backend_pool`: ID was missing the `healthProbeSettings` element

I tested this on terraform v0.12.24 with azurem provider versions v2.13.0 and v2.24.0 with the same results. Perhaps @MelHarbour is right about it being a change in Azure's API?

[lyubomirr]

@kevinchabreck I've had the exact same issue with the same steps as you, but it started working once i changed the provider version to 2.23.0. You can try it.

[MelHarbour]

As a follow-on, we re-ran a previous release on an environment that hadn't been modified outside of Terraform since the last apply, and it also failed with the same error. So it looks distinctly like something's changed in the underlying platform.

[kevinchabreck]

@lyubomirr downgrading to azurerm v2.23.0 seemed to work! Additionally, I had previously modified my resource ID string during the import from .../providers/Microsoft.Network/frontdoors/... to .../providers/Microsoft.Network/frontDoors/... due to an error I got when attempting a previous import. Reverting this and importing the resource ID exactly as it is shown in the Azure console (ie. the one with the lowercase D in frontdoors) fixed this issue. Thanks for the tip!

[MelHarbour]

Did another bit of testing - realised that we were actually using 2.24, so I've also downgraded to 2.23 and it appears to be working again, so looks like the regression is in provider version 2.24.

[MelHarbour]

Presumably the issue was introduced in #8146 as part of the rewriting around IDs.

[tombuildsstuff]

@MelHarbour as you mentioned above, unfortunately the FrontDoor API is broken here in that it's returning these in the incorrect case.

Azure API's are supposed to be case-insensitive for Requests but that URI's listed in Responses should be case-sensitive - the Resource Group can be case-insensitive, but unfortunately the way this has been implemented means that the entire URI can be insensitive at Request time.

Other downstream API's treat URI's in a mixed manner - where some API's require that URI's are case sensitive - and others aren't bothered by casing - as such we intentionally treat them as case-sensitive (since that's how the HTTP spec recommends treating URI's).

Unfortunately the FrontDoor API implements this in such a fashion that they're case-insensitive in the response too (specifically being lower-cased when updated through the Portal) - such that #8146 tries to make this consistent by updating/requiring the ID's to be in a consistent format (since users should be able to rely on these being the same). As such it appears there's some more edge-cases to handle here, as a follow up to #8146 - would anyone be able to provide an example of the ID formats being returned/in the state for these fields?

Thanks!

[andrstor]

@tombuildsstuff I dont know if this is what you are asking, but I have noticed this behaviour:

After creating the frontdoor resource with azurerm v2.24 it looks like this in https://resources.azure.com/.

After editing it manually in the portal (whatever change), it looks like this. Notice how many of the ID's have all become lowercase.

When running terraform plan again now, it fails. You can actually edit the resource in https://resources.azure.com/ manually (edit the template and use PUT), and terraform will start working again (if you managed to correctly adjust all the lowercase ID's.

[tombuildsstuff]

@andrstor perfect, thanks - where the entire ID is being lower-cased, that's a bug in the API (since the health probe names etc should be consistent) - so we'll need to raise an API bug here either way unfortunately

[robselway]

Hi @tombuildsstuff - just to clarify - is the only resolution here to wait for Azure to fix the issue? I'm trying to figure out whether it's worth replacing this resource with another script until it's resolved.

[cpressland]

We fought so hard with Azure Support during some previous Azure Front Door Terraform/API issues to get them to recognise the Azure API was a bit of a mess and provided multiple examples via Terraform, Azure Portal, and Azure CLI. Response was simply that this isn't an issue because the Azure Portal still works, I kinda get it, but I equally don't think it's this projects responsibility to have to constantly build work-arounds to a buggy API. I'll raise this issue with our Account Managers etc again and see if we can get any traction.

[tombuildsstuff]

@robselway

just to clarify - is the only resolution here to wait for Azure to fix the issue?

Based on what I can see, unfortunately yes.

The Azure API Specification states that values should be returned in the casing they're submitted (although the HTTP Specification states URI's should be case-sensitive but I digress) - so unfortunately this is an API bug which needs to be fixed, since this should be returned in the same casing we're submitting it in here.

For what it's worth we've also raised this on our end - unfortunately the Networking API's differ from every other Azure API here, so I don't think we can easily work around this (unless perhaps we can find the original casing from the specific sub-element, but that's assuming the API doesn't change to break the casing there too)

---

@JeffreyRichter this is a good example of the Networking API's returning URI's in a case-insensitive manner which differs from the recommendation in the ARM Spec (which unfortunately differs from the HTTP Specification, where the entire URI is case sensitive, this StackOverflow answer for more details).

Whilst it'd be possible to work around this bug if the "resource type" segment could be parsed case-insensitively, where the entire URI is lower-cased in some responses, but not in others (see this comment for example responses) - there's not much we can do here, since we can't guarantee these are the correct casing (or that the Networking API's won't change this casing in a future update).

Whilst this Github issue is the wrong place for this discussion, I feel like perhaps the current ambiguous behaviour defined in the ARM Specification ("return in the casing the user passed it in as") is the cause of this confusion - perhaps it'd be clearer if the ARM Specification stated that the entire URI must (to use the language of RFC 7230) be treated as case-sensitive in Responses - WDYT?

Thanks!

[tombuildsstuff]

@WodansSon would you be able to get an ETA for a fix from the service team here?

[WodansSon]

@WodansSon would you be able to get an ETA for a fix from the service team here?

@tombuildsstuff I will reach out to the Front Door service team and see how quickly we can get a fix in place for this issue, it will most likely include some cross team collaboration with the portal team to roll back their changes to get this issue totally fixed.

[camallen]

I can confirm the steps mentioned by @kplantus (thank you 👍 ) do work for us on our existing AFD resources. We do not have any WAF configured.

Tested with v2.36.0 of the resource provider and Terraform v0.13.5

It's not ideal to edit the resources directly in the azure portal and I'm not sure what will happen if we edit the AFD resources in the portal again, I assume we might re-break the AFD resource definitions.

Hopefully this is useful for the Azure portal team and helps someone else get TF working again.

[WodansSon]

@camallen Your AFD will be fine now since you corrected the casing that was introduced by the portal lowercase issue. You are safe to edit the Frontdoor via portal as well since they have reverted the changes they made that caused this issue in the first place.

UPDATE: I have just heard back from the portal team and they have confirmed that the lower casing issue also exists in the Frontdoor Web Application Firewall Policies UI layer and they are currently working on a fix for that. I have yet to hear back from the AFD team, but I will keep you posted on the progress of this issue once I receive more information. Thank you. 🚀

[windsurfer123]

We have a very similar problem. We had to upgrade recently to 2.24 because of APIM issues and now when we are trying to manage the existing front door resources we get errors.
Set up: tf either 13.5 or 12.9 azurerm v 2.37
Problem 1: when creating resources terraform sits on the "Still creating ..." for over an hour, at which point it is terminated. All objects are getting created properly. On subsequent runs it want to update the FD, which of course never finishes and has to be terminated.
Problem 2: when trying to import the Frond Door resource created by terraform in Problem 1, we get Error: Error parsing Resource ID ""v ID was missing the frontDoors element
Clarification we have never edited our instances of FD using the portal. After upgrading to azurerm 2.24 (APIM Issues) we had started receiving Error: flattening frontend_endpoint: ID was missing the frontDoorWebApplicationFirewallPolicies element the only way we found working was to destroy the existing FD and rebuild it using the new version of the provider.
Update: tf v12.9 + azurerm v2.37 eventually completes building new FDs. Unlike tf v13.5 + azurerm v2.37 which never completes.

[brunoscota]

same happening here. I had to rollback azurerm version to v2.23.0 as a workaround. Now, I am stuck. ;(

[camallen]

@camallen Your AFD will be fine now since you corrected the casing that was introduced by the portal lowercase issue. You are safe to edit the Frontdoor via portal as well since they have reverted the changes they made that caused this issue in the first place.

Sadly, after editing our FD resources in the Azure portal we now have the same old error and broken TF

Error: flattening routing_rules: flattening frontend_endpoints: ID was missing the frontendEndpoints element`

Looks like the portal is still changing the resource definition from frontendEndpoints to non expected case for the Azure provider.

E.g. in the https://resources.azure.com/ portal this is what I see (with redactions) for one of the broken FD resource routing rules, note the different casing on different routing rules

{
  "id": "/subscriptions/.../resourcegroups/.../providers/Microsoft.Network/Frontdoors/.../FrontendEndpoints/my-custom-domain-org-azurefd-net"
},
{
  "id": "/subscriptions/.../resourcegroups/.../providers/Microsoft.Network/frontdoors/.../frontendendpoints/my-custom-domain"
}

[KyMidd]

Hey @WodansSon and $MSFT internal team, is it possible to provide an ETA for when this collection of resources will be repaired at the API layer? So far we're getting updates after a fix is made but nothing to give us an estimate of how long we'll need to wait for this fix to be implemented. Are we looking at days/weeks/months?

The healthcare teams I support use these resources for security and the time-frame you provide can help inform if we should switch to an alternate IaC solution or other workarounds.

Thanks in advance!
kyler

[WodansSon]

@KyMidd Thank you for your question. I am still attempting to get in contact with the Front Door service team about this and have begun to escalate the issue internally to force some action about getting this fixed. Unfortunately I am not able to provide an ETA at this time, but as soon as I have any new news I will promptly update the status here.

[subesokun]

Somehow I'm now in a kind of dead lock. I need to downgrade to 2.23.0 because of this bug but when I do so I'm running into another frontdoor related TF bug (#8036) which was solved in 2.24.0. Very frustrating.

[WodansSon]

@subesokun I am sorry to hear that, but please be patient, @tombuildsstuff and myself have been working on this issue and I believe we have a solution in the provider which should fix 99.999% if the issues that are currently being hit... We are still testing the solution, but so far it all looks good... again, I am sorry for this pain, but we are doing all we can to correct this issue.

[tombuildsstuff]

Ultimately this'll be fixed via #9750 - which ignores the casing returned from the Azure API and rewrites this to be consistent on Terraform's side, whilst there's downsides to that approach (and the Service Team ultimately need to fix these bugs in the API..) this should workaround this series of API bugs for the moment.

[KyMidd]

@WodansSon and @tombuildsstuff : Sincere thank you from myself and on behalf of the community here for working on a workaround for this issue. I have several teams affected, and despite us pushing hard on Microsoft's internal enterprise support, we haven't made any progress at all. I think we're all aware here that the real root cause of this issue is standard-breaking API implementation on Azure side, and from your user-base I want to say: Thank you.

[ajklotz]

@WodansSon @tombuildsstuff Is there a ticket that we can go and add our voice of support? Anything we can do to help MS understand there is attention from all of us regarding it?

[WodansSon]

@ajklotz I don't believe there is a public facing ticket or support request for this issue. That said, I have included a link to this issue in all internal communications for this problem so they are very much aware of the contention the changes in the API have caused. Thank you for asking!

[camallen]

Thanks @tombuildsstuff & @WodansSon for fixing this 🎩 and thanks to all the folks that contributed to the issue 👍.

At 9th Dec 2020 10:25 UTC it appears that this fix is unreleased https://github.com/terraform-providers/terraform-provider-azurerm/blob/2df9c3193a43380e16f0000f3366b221b31d6c74/CHANGELOG.md#2400-unreleased

Is there a timeline for this fix being officially released? I'm very keen (and I assume so are a lot of folks) to test these fixes and get TF integrations working again.

FWIW I can't see how to use this unreleased provider version via https://www.terraform.io/docs/configuration/provider-requirements.html#version-constraints - maybe I'm missing something....

Any ideas or is it wait till the official release?

[katbyte]

@camallen, this will go out with our weekly release this Thursday (tomorrow)

[KyMidd]

I'm sorry to report that I compiled azurerm provider from master as of this morning and we are seeing the same type of issues on a refresh where terraform's unable to handle.

$ terraform refresh
(removed)
Error: expected "custom_rule.0.match_condition.0.selector" to not be an empty string, got 
  on main.tf line 27, in resource "azurerm_frontdoor_firewall_policy" "frontdoor":
  27: resource "azurerm_frontdoor_firewall_policy" "frontdoor" {
Error: expected "custom_rule.1.match_condition.0.selector" to not be an empty string, got 
  on main.tf line 27, in resource "azurerm_frontdoor_firewall_policy" "frontdoor":
  27: resource "azurerm_frontdoor_firewall_policy" "frontdoor" {

The FrontDoor and FrontDoor WAF rules were created from terraform using the new provider. Note the differing capitalization of "frontDoor" vs "frontdoor". I redacted the account, otherwise unchanged.

/subscriptions/./resourceGroups/TestingKyler/providers/Microsoft.Network/frontDoorWebApplicationFirewallPolicies/KylerTestingWafPolicy

And from the portal:

/subscriptions/./resourceGroups/TestingKyler/providers/Microsoft.Network/frontdoorWebApplicationFirewallPolicies/KylerTestingWafPolicy

If possible, I'd like to see this issue reopened. Thanks team.

[WodansSon]

@KyMidd Sorry that you are hitting that issue, however the casing should not matter anymore, I believe you are hitting a validation rule. Do you have a repro for this? If I can get a clear repro for this issue I might be able to get a fix in before the next release. Thank you.

I just looked at the code, what appears to be going on is that you have selector = "" in the config file, which would trigger the below validation rule. Can you confirm if this is the case or not in your config file?

"selector": {
	Type:         schema.TypeString,
	Optional:     true,
	ValidateFunc: validation.StringIsNotEmpty,
},

[KyMidd]

Hey @WodansSon , Oh, I assumed Terraform was modifying the casing on import.

As for repro, definitely.

1. Using this config, simple WAF + FrontDoor: https://github.com/KyMidd/azurerm-frontdoor-testing-repro
2. Compile from master azurerm and store properly
3. Authenticate to Azure via az login
4. terraform init (should find local azurerm)
5. terraform apply <-- Resources built properly
6. terraform refresh <-- Terraform fails with error message

Let me know if I can help further by testing a PR. Thank you!

[KyMidd]

@WodansSon : Update, I have destroyed and recreated this same config several times now to make sure the same error is generated, and I'm unable to replicate it. I'm not sure what happened there. I am now seeing FrontDoor reliably managed by terraform!

[WodansSon]

@KyMidd, I am unable to repro the behavior you are reporting above. That would explain my results as well! Awesome! 🚀

I followed your steps with the same config file and when I execute step 6 I get:

Refresh

azurerm_resource_group.testing_kyler: Refreshing state... [id=/subscriptions/{subscription}/resourceGroups/XXXXXX-frontDoor-Repro]
azurerm_frontdoor_firewall_policy.frontdoor: Refreshing state... [id=/subscriptions/{subscription}/resourceGroups/XXXXXX-frontDoor-Repro/providers/Microsoft.Network/frontDoorWebApplicationFirewallPolicies/KylerTestingWafPolicy]
azurerm_frontdoor.example: Refreshing state... [id=/subscriptions/{subscription}/resourceGroups/XXXXXX-frontDoor-Repro/providers/Microsoft.Network/frontDoors/XXXXXX-testing-frontdoor]

Plan

Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.

azurerm_resource_group.testing_kyler: Refreshing state... [id=/subscriptions/{subscription}/resourceGroups/XXXXXX-frontDoor-Repro]
azurerm_frontdoor_firewall_policy.frontdoor: Refreshing state... [id=/subscriptions/{subscription}/resourceGroups/XXXXXX-frontDoor-Repro/providers/Microsoft.Network/frontDoorWebApplicationFirewallPolicies/KylerTestingWafPolicy]
azurerm_frontdoor.example: Refreshing state... [id=/subscriptions/{subscription}/resourceGroups/XXXXXX-frontDoor-Repro/providers/Microsoft.Network/frontDoors/XXXXXX-testing-frontdoor]

------------------------------------------------------------------------

No changes. Infrastructure is up-to-date.

This means that Terraform did not detect any differences between your
configuration and real physical resources that exist. As a result, no
actions need to be performed.

[ghost]

This has been released in version 2.40.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.40.0"
}
# ... other configuration ...

[jacksondaw]

This issue was still occurring for me when I added a second frontend. I tried upgrading to 2.40.0 and downgrading to 2.23.0, both were throwing this error in relation to the existing frontend.

Error: updating Custom HTTPS configuration for Frontend Endpoint "*" (Front Door "" / Resource Group "FrontDoor"): unable to enable/update Custom Domain HTTPS for Frontend Endpoint "" (Resource Group "FrontDoor"): enabling Custom Domain HTTPS for Frontend Endpoint: frontdoor.FrontendEndpointsClient#EnableHTTPS: Failure sending request: StatusCode=400 -- Original Error: Code="BadRequest" Message="That action isn’t allowed in this profile."

I ended up pulling the https configuration out into the frontdoor_custom_https_configuration and error still throws, but the resources are created?

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!