Creation of management group is failed when using azurerm with Service Principal authentication schema due to 403 error in GET request of management group after received its "Succeeded" status

[oleksandrkudin]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureRM Provider) Version

Terraform v0.12.26
+ provider.azurerm v2.15.0

Affected Resource(s)

• azurerm_management_group

Terraform Configuration Files

provider "azurerm" {
  version         = "~> 2"
  features {}
}

resource "azurerm_management_group" "top_mg" {
  name                          = "dr11-top-mg"
}

resource "azurerm_management_group" "h_mg" {
  name                          = "dr11-h-mg"
  parent_management_group_id    = azurerm_management_group.top_mg.id
}

resource "azurerm_management_group" "p_mg" {
  name                          = "dr11-p-mg"
  parent_management_group_id    = azurerm_management_group.top_mg.id
}

resource "azurerm_management_group" "t_mg" {
  name                          = "dr11-t-mg"
  parent_management_group_id    = azurerm_management_group.top_mg.id
}

resource "azurerm_management_group" "d_mg" {
  name                          = "dr11-d-mg"
  parent_management_group_id    = azurerm_management_group.top_mg.id
}

Debug Output

https://gist.github.com/oleksandrkudin/d7fb8026797bfde5287a54617ba4eb8a

Panic Output

Expected Behavior

Management Group is created and resource added to Terraform state

Actual Behavior

Management Group is created in Azure as resource as well as "Succeeded" creation status received successfully but after this last GET HTTP request perfomed by azurerm for final verification is failed with 403 (Forbidden) code and so whole management group creation process is failed with next Terraform error.

Terraform error:

Error: unable to retrieve Management Group "dr11-h-mg": managementgroups.Client#Get: Failure responding to request: StatusCode=403 -- Original Error: autorest/azure: Service returned an error. Status=403 Code="AuthorizationFailed" Message="The client '3e47cd00-0bba-40fd-b6a5-31bf8005baba' with object id '3e47cd00-0bba-40fd-b6a5-31bf8005baba' does not have authorization to perform action 'Microsoft.Management/managementGroups/read' over scope '/providers/Microsoft.Management/managementGroups/dr11-h-mg' or the scope is invalid. If access was recently granted, please refresh your credentials."

Fragment from DEBUG (sucessfull "Succeeded" creation status + 403 error code for GET request after this)

....
2020-06-24T10:58:44.956+0300 [DEBUG] plugin.terraform-provider-azurerm_v2.15.0_x5.exe: [DEBUG] AzureRM Response for https://management.azure.com/providers/Microsoft.Management/operationResults/create/managementGroups/dr11-h-mg?api-version=2018-03-01-preview: 
2020-06-24T10:58:44.956+0300 [DEBUG] plugin.terraform-provider-azurerm_v2.15.0_x5.exe: HTTP/2.0 200 OK

2020-06-24T10:58:44.956+0300 [DEBUG] plugin.terraform-provider-azurerm_v2.15.0_x5.exe: {"id":"/providers/Microsoft.Management/managementGroups/dr11-h-mg","type":"/providers/Microsoft.Management/managementGroups","name":"dr11-h-mg","status":"Succeeded","properties":{"tenantId":"3cf6f2e2-f8fa-45ac-907f-748b18866f47","displayName":"dr11-h-mg","details":{"version":18,"updatedTime":"2020-06-24T07:58:27.4579916Z","updatedBy":"3e47cd00-0bba-40fd-b6a5-31bf8005baba","parent":{"id":"/providers/Microsoft.Management/managementGroups/dr11-top-mg","name":"dr11-top-mg","displayName":"dr11-top-mg"}}}}
....

2020-06-24T10:58:44.956+0300 [DEBUG] plugin.terraform-provider-azurerm_v2.15.0_x5.exe: GET /providers/Microsoft.Management/managementGroups/dr11-h-mg?%24expand=children&%24recurse=false&api-version=2018-03-01-preview HTTP/1.1
2020-06-24T10:58:44.956+0300 [DEBUG] plugin.terraform-provider-azurerm_v2.15.0_x5.exe: Host: management.azure.com
...
2020-06-24T10:58:45.307+0300 [DEBUG] plugin.terraform-provider-azurerm_v2.15.0_x5.exe: [DEBUG] AzureRM Response for https://management.azure.com/providers/Microsoft.Management/managementGroups/dr11-h-mg?%24expand=children&%24recurse=false&api-version=2018-03-01-preview: 
2020-06-24T10:58:45.307+0300 [DEBUG] plugin.terraform-provider-azurerm_v2.15.0_x5.exe: HTTP/2.0 403 Forbidden

Making manual GET https://management.azure.com/providers/Microsoft.Management/operationResults/create/managementGroups/dr11-h-mg?api-version=2018-03-01-preview request by using last received access_token from Terraform DEBUG output returns 200 OK and management group description.

Steps to Reproduce

1. Assign service principal as owner of Root Management Group
2. Set proper local env variables to connect with SP:

export ARM_SUBSCRIPTION_ID="xxxxxx"
export ARM_CLIENT_ID="xxxxxx"
export ARM_CLIENT_SECRET="xxxxx"
export ARM_TENANT_ID="xxxxx"

3. terraform init
4. terraform apply

Important Factoids

• No issue with azurerm 1.44.0 provider. Issue is noticed after upgrading to 2.14.0, 2.15.0 azurerm providers
• azurerm authentication with Service Principal
• Issue is intermittent and not permanent. Sometimes creation of azurerm_management_group.top_mg is failed, but sometimes azurerm_management_group.top_mg is created but one of nested to top_mg management group is failed. 1 of 5 terraform apply attempts may be successful without any errors.

References

• Azure Management Group creation with Service Principal returns 403 #6091

[ghost]

This has been released in version 2.16.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.16.0"
}
# ... other configuration ...

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!