IoT Hub Consumer Group creation fails if the same 'apply' step tries to create a component, which depends on it

[v1r7u]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

Azure IoT Hub supports built-in event-hub endpoint, where device-telemetry messages are routed by default. Terraform supports adding Consumer Groups to this endpoint. Other parts of terraform configuration should be able to use Consumer Group resource as a reference.

We use a setup, when Azure Function reads telemetry data from IoT Hub, using a dedicated consumer group and IoT Hub policy. When this infrastructure is created the first time, terraform throws an exception like:

Error: Error retrieving Consumer Group "normalizer" (Endpoint "events" / IoTHub "iothub-name" / Resource Group "rg-name"): devices.IotHubResourceClient#GetEventHubConsumerGroup: Failure sending request:
StatusCode=409 -- Original Error: autorest/azure: Service returned an error. Status=<nil> <nil>

The problem appears if IoT Hub and Consumer Group resources are created in the same apply step with the dependent component.

Terraform (and AzureRM Provider) Version

Terraform v0.12.26
+ provider.azurerm v2.15.0

Affected Resource(s)

• azurerm_iothub_consumer_group

Terraform Configuration Files

The entire configuration is committed to the repo: https://github.com/v1r7u/azurerm_issues/blob/master/src/iot_eventhub_listener/main.tf

Expected Behavior

Terraform is able to create in the same step: IoT Hub, IoT Hub Policy, IoT Hub Consumer Group, and Azure Function which consumes them.

Actual Behavior

Terraform properly creates IoT Hub and IoT Hub Policy, fails during creating Consumer Group and Azure Function, which depends on it.

Steps to Reproduce

Detailed steps to reproduce and terraform configuration sample I've committed to the repo:
https://github.com/v1r7u/azurerm_issues/tree/master/src/iot_eventhub_listener

[royto]

We had the same kind of issue on iothub when trying to create in the same apply 2 consumer group and 1 SAS Policy.

As a workaround, we set an explicit dependency with depends_on and it has fix our pb.

[v1r7u]

@royto, thanks for forcing me into this direction!
I did a few more tests and found that in my case azurerm_iothub_consumer_group creation fails, likely, because of explicitly defined default IoT Hubroute.

Below are simplified version of my resources (I dropped registration of other endpoints/routes to shorten the sample)

What I had before (this one fails all the time):

resource "azurerm_iothub" "ingestion" {
  name                = var.iothub_name
  resource_group_name = azurerm_resource_group.rg_ingestion.name
  location            = azurerm_resource_group.rg_ingestion.location

  sku {
    name     = var.iot_sku.name
    capacity = var.iot_sku.capacity
  }

  route {
    name           = "defaultroute"
    source         = "DeviceMessages"
    condition      = "true"
    endpoint_names = ["events"]
    enabled        = true
  }
}

resource "azurerm_iothub_consumer_group" "normalizer" {
  name                   = "normalizer"
  iothub_name            = azurerm_iothub.ingestion.name
  eventhub_endpoint_name = "events"
  resource_group_name    = azurerm_resource_group.rg_ingestion.name
}

How I changed it (this one works fine most of the time):

resource "azurerm_iothub" "ingestion" {
  name                = var.iothub_name
  resource_group_name = azurerm_resource_group.rg_ingestion.name
  location            = azurerm_resource_group.rg_ingestion.location

  sku {
    name     = var.iot_sku.name
    capacity = var.iot_sku.capacity
  }
}

resource "azurerm_iothub_route" "default_iot_eh_route" {
  resource_group_name = azurerm_resource_group.rg_ingestion.name
  iothub_name         = azurerm_iothub.ingestion.name
  name                = "defaultroute"

  source         = "DeviceMessages"
  condition      = "true"
  endpoint_names = ["events"]
  enabled        = true
}

resource "azurerm_iothub_consumer_group" "normalizer" {
  name                   = "normalizer"
  iothub_name            = azurerm_iothub.ingestion.name
  eventhub_endpoint_name = "events"
  resource_group_name    = azurerm_resource_group.rg_ingestion.name

  depends_on = [ azurerm_iothub_route.default_iot_eh_route ]
}

Root cause

If the default route for events endpoint is explicitly stated inside the azurerm_iothub resource, dependent consumer group fails to create. I have a feeling, that defining this route causes a background processes on azure side and API acknowledges successful response before the process is completed.

I do not fully understand, how GET request might return 409 Status Code, but I can not find any documentation about the matter.

The failure happens in resourceArmIotHubConsumerGroupCreate function, which calls azure-sdk GetEventHubConsumerGroup method after creating the consumer group.

Workaround

Move default route definition out of azurerm_iothub and define azurerm_iothub_consumer_group explicit dependency on the route (see example above) increases the probability of successful operation, but still might fail.

As a hack, I added the consumer-group dependency on iot-hub-policy, which takes a while to complete. It's a kind of simulated delay, but I've just gave up to debug the issue further.

[royto]

Glad it helps 😉

[neil-yechenwei]

Thanks for opening this issue. After investigated, I assume iothub seems api doesn't allow multiple operations in the same time. So I made a fix to add a lock on iothub while operating iothub consumer group. Hopes it would be helpful. Thanks.

[ghost]

This has been released in version 2.23.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.23.0"
}
# ... other configuration ...

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!