Error: Provider produced inconsistent result after apply

[pankajnarang]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritise this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritise the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform (and AzureAD Provider) Version

Affected Resource(s)

• azuread_invitation + azuread_group_member

Terraform Configuration Files

resource "azuread_invitation" "User" {
  user_display_name  = "Display Name"
  user_email_address = "[email protected]"
  redirect_url       = "https://portal.azure.com"

  message {
    additional_recipients = ["[email protected]"]
    body                  = "Hello there! You are invited to join my Azure tenant!"
  }
}

resource "azuread_group_member" "example" {
  group_object_id  = "XXXXXXXXX"
  member_object_id = azuread_invitation.User.user_id
}

Debug Output

╷
│ Error: Provider produced inconsistent result after apply
│
│ When applying changes to azuread_group_member.example, provider "provider["registry.terraform.io/hashicorp/azuread"]" produced an unexpected new value: Root resource was present, but now absent.
│
│ This is a bug in the provider, which should be reported in the provider's own issue tracker.

Expected Behavior

azuread_invitation.User: Creating...
azuread_invitation.User: Creation complete after 3s [id=xxxx]
azuread_group_member.example: Creating...
azuread_group_member.example: Creation complete after 1s [id=xxxx]

Steps to Reproduce

1. terraform apply

Important Factoids

The issue is not consistent, and the script works as expected if we run the same twice but still throws error in the output

[manicminer]

Thanks for reporting this @pankajnarang. I suspect this may be due to eventual consistency issues in Azure AD since creating invitations is slower to reflect new users than just creating a new user directly. Will take a closer look and try to resolve this.

[manicminer]

@pankajnarang Please can you provide a debug trace that shows the API requests involved when this occurs? This will greatly help to diagnose the source of the error. Thanks!

[lukasgabi]

We are having the same issue with azure_ad_group resources, it appears intermittently. I'm assuming it is the same issue, but it's difficult to replicate as it doesn't happen regularly. We are deploying and destroying our infrastructure on schedule every two hours, that's how we noticed in the first place, as it would fail once we receive this error on terraform apply. We will enable debug and report back once it happens again.

[lukasgabi]

I managed to reproduce the error, I attached the logs created with TF_LOG=debug, TF version 1.0.9 and azuread provider version 2.8.0
tf.log

[maxhillaert]

I think we are hitting the same issue:

When applying changes to
azuread_app_role_assignment.blablabla["b0e81d5a-2c6d-495e-822c-402d457779c2"],
provider "provider["registry.terraform.io/hashicorp/azuread"]" produced an
unexpected new value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

Running again later fails.

Error: Could not create app role assignment

with azuread_app_role_assignment.blablabl["b0e81d5a-2c6d-495e-822c-402d457779c2"],
on blablabl.tf line 1, in resource "azuread_app_role_assignment" "blablabl":
1: resource "azuread_app_role_assignment" "blablabl" {

AppRoleAssignedToClient.BaseClient.Post(): unexpected status 400 with OData
error: Request_BadRequest: Permission being assigned already exists on the
object

terraform {
required_version = "0.15.5"
required_providers {
azurerm = {
version = ">= 2.73.0"
}
azuread = {
version = "1.5.0"
}
}
}

[ppanyukov]

So @manicminer is my understanding correct that it fails here?

# Create group membership, 2021-11-02T14:11:18.141Z
POST /beta/2231b15c-2ed7-4875-8107-a60871307a98/groups HTTP/1.1

# Immediately trying to get this back fails with 404, 2021-11-02T14:11:18.148Z
GET https://graph.microsoft.com/beta/2231b15c-2ed7-4875-8107-a60871307a98/groups/3959121a-3cc7-4c36-b20e-1420bf8e7688

So it's nothing to do with the fact that the users are invited?

If this is the case, then the workaround would be in the provider to retry/delay or similar?

Incidentally we've just hit this too, and of course running terraform apply again fails because group membership already exists, and to fix this we need to do one of:

1. Import the failed group memberships into the state. OR
2. Delete group membership manually in AAD and apply again.

Neither are pretty so would be great to have a fix!

---

EDIT1

Actually I'm not sure the race is in group membership. We use it quite a bit for existing users and never seen this issue there. It only cropped up with invites. So it maybe some propagation delay with invites after all? Also weirdly enough the group membership does get created, at least in all cases for us so far. This Graph API is sooo confusing :)

[manicminer]

@maxhillaert That looks like a separate bug for the azuread_app_role_assignment resource, please could you raise this as a new issue and post a debug log if you can? Thanks!

[manicminer]

@ppanyukov It's almost certainly an eventual consistency issue due to the distributed architecture of Azure AD. We try to handle this as best we can with retries, but it's complicated by the fact that the API often returns a valid response long before an object is fully replicated. This makes it impossible in some cases for us to reliably tell whether a new resource is completed and ready to use. Our strategy is generally to look for X number of confirmatory responses but even this isn't always reliable.

Invitations are particularly guilty of this, and because they're write-only it adds to the difficulty in determining whether a newly invited user is ready to be used, say to add them to a group. I'll do some investigations over the next few days and try to narrow this down further.

[ppanyukov]

@manicminer and all others, since this is the propagation delay in the invited users, the workaround I've tested and found works well is to use time_sleep, like in the following example (we use loops and it works there too). Hope it helps everyone looking for a temporary workaround while waiting for the provider patch.

# Simple example of waiting for invited user propagation before
# applying group membership.
resource "azuread_group" "example_simple" {
  display_name     = "example-simple"
  security_enabled = true
}

resource "azuread_invitation" "example_simple" {
  user_display_name  = "[email protected]"
  user_email_address = "[email protected]"
  redirect_url       = "https://portal.azure.com"
}

resource "time_sleep" "example_simple_propagation" {
  create_duration  = "60s" # should be enough right?
  destroy_duration = "0s"

  triggers = {
    azuread_invitation = azuread_invitation.example_simple.user_id
  }
}

resource "azuread_group_member" "example_simple" {
  group_object_id  = azuread_group.example_simple.object_id
  member_object_id = azuread_invitation.example_simple.user_id

  # Wait for all users to propagate before applying group membership
  depends_on = [
    time_sleep.example_simple_propagation,
  ]
}

Ditto with loops, just in case:

# ===========
# Same example but with loops.

resource "azuread_group" "example_loop" {
  display_name     = "example-loop"
  security_enabled = true
}


locals {
  users_external = [
    {
      user_display_name  = "[email protected]"
      user_email_address = "[email protected]"
    }
  ]
}

resource "azuread_invitation" "example_loop" {
  for_each = {
    for v in local.users_external : v.user_email_address => v
  }

  user_display_name  = each.value.user_display_name
  user_email_address = each.value.user_email_address
  redirect_url       = "https://portal.azure.com"
}

resource "time_sleep" "example_loop_propagation" {
  for_each = azuread_invitation.example_loop

  create_duration  = "30s"
  destroy_duration = "0s"

  triggers = {
    azuread_invitation = azuread_invitation.example_loop[each.key].user_id
  }
}

resource "azuread_group_member" "example_loop" {
  for_each = azuread_invitation.example_loop

  group_object_id  = azuread_group.example_loop.object_id
  member_object_id = azuread_invitation.example_loop[each.key].user_id

  # Wait for all users to propagate before applying group membership
  depends_on = [
    time_sleep.example_loop_propagation,
  ]
}

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.