Support for Azure Event Hub - Log compaction Feature

[ValentinVers]

Is there an existing issue for this?

• I have searched the existing issues

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

When creating an Azure Event Hub cleanup policy can be set to "Delete" with a particular retention time or to "Compact" with a Tombstone retention time.

This new feature is currently not supported in azurerm_eventhub.

New or Affected Resource(s)/Data Source(s)

azurerm_eventhub

Potential Terraform Configuration

resource "azurerm_eventhub" "example" {
  name                = "acceptanceTestEventHub"
  namespace_name      = azurerm_eventhub_namespace.example.name
  resource_group_name = azurerm_resource_group.example.name
  partition_count     = 2
  cleanup_policy      = "Compact" // or "Delete"
  tombstone_retention = 1 // In hours
}

References

• https://learn.microsoft.com/en-us/azure/event-hubs/log-compaction
• https://learn.microsoft.com/en-us/rest/api/eventhub/preview/event-hubs/create-or-update?tabs=HTTP

[repcaks]

Any updates or planned delivery date?

[dico-harigkev]

This feature is still missing and would be important for a use case we have.

Also, trying to import an Azure EventHub instance (azurerm_eventhub) which has been created with a cleanup policy of "Compact" through the Azure Portal even crashes the azurerm provider, because in that case the API seems to return a value for message_retention that can not be parsed as integer (too large, > 2^63):

╷
│ Error: Request cancelled
│ 
│   with azurerm_eventhub.example,
│   on example.tf line 31, in resource "azurerm_eventhub" "example":
│   31: resource "azurerm_eventhub" "example" {
│ 
│ The plugin.(*GRPCProvider).ReadResource request was cancelled.
╵

Stack trace from the terraform-provider-azurerm_v3.113.0_x5 plugin:

panic: Error reading level state: strconv.ParseInt: parsing "9223372036854776000": value out of range

goroutine 202 [running]:
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*ResourceData).get(0xc003202180, {0xc0032012e0, 0x1, 0x1}, 0x8?)
        github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource_data.go:556 +0x333
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*ResourceData).State(0xc003202180)
        github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource_data.go:354 +0x473
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*Resource).ShimInstanceStateFromValue(0xc000cffc00, {{{0x8a58448?, 0xc000631790?}}, {0x7306ac0?, 0xc003206c60?}})
        github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/resource.go:670 +0x1d8
github.com/hashicorp/terraform-plugin-sdk/v2/helper/schema.(*GRPCProviderServer).ReadResource(0xc000413b90, {0x8a558c0?, 0xc003206360?}, 0xc0031dc840)
        github.com/hashicorp/terraform-plugin-sdk/[email protected]/helper/schema/grpc_provider.go:623 +0x1f3
github.com/hashicorp/terraform-plugin-go/tfprotov5/tf5server.(*server).ReadResource(0xc000aa7220, {0x8a558c0?, 0xc0031fbb90?}, 0xc003198b40)
        github.com/hashicorp/[email protected]/tfprotov5/tf5server/server.go:789 +0x48b
github.com/hashicorp/terraform-plugin-go/tfprotov5/internal/tfplugin5._Provider_ReadResource_Handler({0x7ed90e0?, 0xc000aa7220}, {0x8a558c0, 0xc0031fbb90}, 0xc000ab5490, 0x0)
        github.com/hashicorp/[email protected]/tfprotov5/internal/tfplugin5/tfplugin5_grpc.pb.go:431 +0x169
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000038000, {0x8a805c0, 0xc001c041a0}, 0xc0031fd440, 0xc001bd69c0, 0xe081468, 0x0)
        google.golang.org/[email protected]/server.go:1374 +0xde7
google.golang.org/grpc.(*Server).handleStream(0xc000038000, {0x8a805c0, 0xc001c041a0}, 0xc0031fd440, 0x0)
        google.golang.org/[email protected]/server.go:1751 +0x9e7
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        google.golang.org/[email protected]/server.go:986 +0xbb
created by google.golang.org/grpc.(*Server).serveStreams.func1 in goroutine 52
        google.golang.org/[email protected]/server.go:997 +0x145

Error: The terraform-provider-azurerm_v3.113.0_x5 plugin crashed!

This is always indicative of a bug within the plugin. It would be immensely
helpful if you could report the crash with the plugin's maintainers so that it
can be fixed. The output above should help diagnose the issue.

[rutgersc]

Last time I've looked at this (btw I don't know what I'm doing), there was some mismatch betweeen the go-azure-sdk api version and the actual api versions deployed by azure : hashicorp/go-azure-sdk#976. There has been some update, so maybe it's fixed now.

This is what I had been working with: main...rutgersc:terraform-provider-azurerm:main2.

[deichrenner]

We're in need for this feature as well. @rutgersc do you have plans on opening a PR with your changes?

[rutgersc]

We're in need for this feature as well. @rutgersc do you have plans on opening a PR with your changes?

No, not yet. Running into the same issue as dico-harigkev, where it's unable to parse into an int panic: Error reading level state: strconv.ParseInt: parsing "9223372036854776000": value out of range

This now also applies to the new RetentionDescription.RetentionTimeInHours:

If cleanupPolicy is Compact the returned value of this property is Long.MaxValue.

Not sure what to do here. Of the available schema types, only int is currently available: https://github.com/hashicorp/terraform-provider-azurerm/blob/main/internal/tf/pluginsdk/schema.go. Maybe someone familiar with the codebase can weigh in on this? @tombuildsstuff?

This branch is able to create the eventhub with the correct cleanup policy.

[rickardp]

We're in need for this feature as well. @rutgersc do you have plans on opening a PR with your changes?

No, not yet. Running into the same issue as dico-harigkev, where it's unable to parse into an int panic: Error reading level state: strconv.ParseInt: parsing "9223372036854776000": value out of range

This now also applies to the new RetentionDescription.RetentionTimeInHours:

If cleanupPolicy is Compact the returned value of this property is Long.MaxValue.

Not sure what to do here. Of the available schema types, only int is currently available: https://github.com/hashicorp/terraform-provider-azurerm/blob/main/internal/tf/pluginsdk/schema.go. Maybe someone familiar with the codebase can weigh in on this? @tombuildsstuff?

This branch is able to create the eventhub with the correct cleanup policy.

What happened with this PR? Any input from maintainers?

[rutgersc]

What happened with this PR? Any input from maintainers?

No idea, there isn't a PR yet. I likely won't spend more time on this either, maybe someone else can pick this up.