Failing to create EventHub > 32 partition through Terraform

[meetamitbhatia]

Sometimes, EventHub Product Group (PG) considers exception to allow creating EventHub with Partition count >32 i.e. upto 40 or for some exceptional cases upto 64. Once exception is applied, EventHub can be created with >32 partitions i.e. upto 40 or 64 through Powershell/CLI/REST API.

Recently, one of the top Azure customers had a same requirement to increase partition quota and it was increased by EventHub PG to 64. Now, this customer uses Terraform script to create EventHub. When they tried creating EventHub with partition >32 , Terraform script failed throwing error: "Error: EventHub Partition Count has to be between 1 and 32”
So, they had to use another method to create EventHub i.e. using Powershell/CLI.

It seems Terraform has validation to check partition count between 1 and 32,

ValidateEventHubPartitionCount function that limits the partition count to no more than 32. The fix would be simple, just update 32 to be 64… in most cases when this happens it is because the documentation states that the limit is something, in this case 32 partitions, but a particular subscription has had an exception granted by the service team.

       if !(32 >= value && value >= 1) {
                  errors = append(errors, fmt.Errorf("EventHub Partition Count has to be between 1 and 32"))
       }

There is still a bug in this code, the if statement should be 32 > value not 32 >= value, the latter will cause an error if you have 32 partitions; effectively limiting the resource to 31 partitions.

They’re requesting if there’re plans to change this validation. Could you please confirm if validation can be fixed?

Please find the debug logs for the same operation,

2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_resource_group.metrics" references: []
2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_template_deployment.kusto_db" references: []
2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_storage_account.auto_scale_logs[0]" references: []
2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_storage_account.auto_scale_logs[1]" references: []
2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_eventhub_namespace.metrics[0]" references: []
2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_eventhub_namespace.metrics[1]" references: []
2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_eventhub.log[0]" references: []
2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_eventhub.log[1]" references: []
2020/06/08 23:54:25 [WARN] Provider "registry.terraform.io/-/azurerm" produced an invalid plan for module.metrics.azurerm_template_deployment.kusto_db, but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .parameters: planned value cty.NullVal(cty.Map(cty.String)) does not match config value cty.MapValEmpty(cty.String)
2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_eventhub.user[1]" references: []
2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_eventhub.user[0]" references: []

Warning: Resource targeting is in effect

2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_template_deployment.kusto_connections[0]" references: []
2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_template_deployment.kusto_connections[1]" references: []
You are creating a plan with the -target option, which means that the result
2020/06/08 23:54:25 [ERROR] module.metrics: eval: *terraform.EvalDiff, err: EventHub Partition Count has to be between 1 and 32
2020/06/08 23:54:25 [ERROR] module.metrics: eval: *terraform.EvalSequence, err: EventHub Partition Count has to be between 1 and 32
2020/06/08 23:54:25 [ERROR] module.metrics: eval: *terraform.EvalDiff, err: EventHub Partition Count has to be between 1 and 32
2020/06/08 23:54:25 [ERROR] module.metrics: eval: *terraform.EvalSequence, err: EventHub Partition Count has to be between 1 and 32
2020/06/08 23:54:25 [WARN] Provider "registry.terraform.io/-/azurerm" produced an invalid plan for module.metrics.azurerm_storage_account.auto_scale_logs[0], but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .network_rules: block count in plan (1) disagrees with count in config (0)
- .queue_properties: block count in plan (1) disagrees with count in config (0)
- .blob_properties: block count in plan (1) disagrees with count in config (0)
2020/06/08 23:54:25 [WARN] Provider "registry.terraform.io/-/azurerm" produced an invalid plan for module.metrics.azurerm_storage_account.auto_scale_logs[1], but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .queue_properties: block count in plan (1) disagrees with count in config (0)
- .blob_properties: block count in plan (1) disagrees with count in config (0)
- .network_rules: block count in plan (1) disagrees with count in config (0)
of this plan may not represent all of the changes requested by the current
20/06/08 23:54:25 [WARN] Provider "registry.terraform.io/-/azurerm" produced an invalid plan for module.metrics.azurerm_template_deployment.kusto_connections[1], but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .parameters: planned value cty.NullVal(cty.Map(cty.String)) does not match config value cty.MapValEmpty(cty.String)
2020/06/08 23:54:25 [WARN] Provider "registry.terraform.io/-/azurerm" produced an invalid plan for module.metrics.azurerm_template_deployment.kusto_connections[0], but we are tolerating it because it is using the legacy plugin SDK.
The following problems may be the cause of any confusing errors from downstream operations:
- .parameters: planned value cty.NullVal(cty.Map(cty.String)) does not match config value cty.MapValEmpty(cty.String)
configuration.

2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_monitor_diagnostic_setting.auto_scale_logs_diagnostic[0]" references: []
2020/06/08 23:54:25 [DEBUG] ReferenceTransformer: "module.metrics.azurerm_monitor_diagnostic_setting.auto_scale_logs_diagnostic[1]" references: []
2020/06/08 23:54:25 [INFO] backend/local: plan operation completed
The -target option is not for routine use, and is provided only for
exceptional situations such as recovering from errors or mistakes, or when
Terraform specifically suggests to use it as part of an error message.

Warning: "kafka_enabled": [DEPRECATED] This field is now automatically set depending on the SKU used - as such it's no longer used and will be removed in 2.0 of the Azure Provider

on src/modules/metrics/eventhub.tf line 13, in resource "azurerm_eventhub_namespace" "metrics":
13: resource "azurerm_eventhub_namespace" "metrics" {

Error: EventHub Partition Count has to be between 1 and 32

on src/modules/metrics/eventhub.tf line 26, in resource "azurerm_eventhub" "log":
26: resource "azurerm_eventhub" "log" {

Error: EventHub Partition Count has to be between 1 and 32

on src/modules/metrics/eventhub.tf line 51, in resource "azurerm_eventhub" "user":
51: resource "azurerm_eventhub" "user" {

2020-06-08T23:54:25.776-0700 [DEBUG] plugin: plugin process exited: path=/Users/abc/dev/excelerate-ops/.terraform/plugins/darwin_amd64/terraform-provider-azurerm_v1.44.0_x4 pid=18309
2020-06-08T23:54:25.776-0700 [DEBUG] plugin: plugin exited
2020-06-08T23:54:25.776-0700 [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform pid=18257
2020-06-08T23:54:25.776-0700 [DEBUG] plugin: plugin process exited: path=/usr/local/bin/terraform pid=18306
2020-06-08T23:54:25.776-0700 [DEBUG] plugin: plugin exited
2020-06-08T23:54:25.776-0700 [DEBUG] plugin: plugin exited

[neil-yechenwei]

@meetamitbhatia, Thanks for opening this issue. Seems if you want to create eventhub with >32 partitions, you have to use azurerm_eventhub_cluster to implement it and the upper limit for the validation in azurerm_eventhub has been increased in PR but it hasn't been released. I assume it would be available in next version. This is the example of the usage if you need.

[ghost]

This has been released in version 2.18.0 of the provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading. As an example:

provider "azurerm" {
    version = "~> 2.18.0"
}
# ... other configuration ...

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. If you feel I made an error 🤖 🙉 , please reach out to my human friends 👉 [email protected]. Thanks!