aws_kinesis_firehose_delivery_stream to support Dynamic Partitioning

[jsyrjala]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

AWS Kinesis Firehose has a new feature Dynamic Partitioning. There doesn't seem to be support for this feature. At least the Terraform documentation for kinesis_firehose_delivery_stream doesn't mention it.

With Dynamic Partitioning, you can continuously partition streaming data in Kinesis Data Firehose using keys within data like “customer_id” or “transaction_id” and deliver data grouped by these keys into corresponding Amazon Simple Storage Service (Amazon S3) prefixes, making it easier for you to run high performance, cost-efficient analytics on streaming data in Amazon S3 using Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum.

New or Affected Resource(s)

• aws_kinesis_firehose_delivery_stream

Potential Terraform Configuration

# Copy-paste your Terraform configurations here - for large Terraform configs,
# please use a service like Dropbox and share a link to the ZIP file. For
# security, you can also encrypt the files using our GPG public key.

References

• https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/kinesis_firehose_delivery_stream
• https://aws.amazon.com/about-aws/whats-new/2021/08/introducing-dynamic-partitioning-amazon-kinesis-data-firehose/

[avegao]

With these options (processing_configuration block) I can "manage" the delivery stream, but whit limitations:

• I can't deploy a new delivery stream: Error: error creating Kinesis Firehose Delivery Stream: InvalidArgumentException: Dynamic Partitioning Namespaces can only be part of a prefix expression when Dynamic Partitioning is enabled. I need to do an import creating the delivery stream from AWS web console.
• I can't update any delivery stream with Dynamic Partitioning enabled.

resource "aws_kinesis_firehose_delivery_stream" "fh" {

  extended_s3_configuration {
      role_arn            = aws_iam_role.role.arn
      bucket_arn          = var.s3_bucket_arn
      buffer_interval     = 900
      buffer_size         = 128
      compression_format  = "UNCOMPRESSED"
      prefix              = "clientId=!{partitionKeyFromQuery:clientId}/"
      error_output_prefix = "error/"
  
      processing_configuration {
        enabled = "true"
  
        processors {
          type = "MetadataExtraction"
  
          parameters {
            parameter_name  = "MetadataExtractionQuery"
            parameter_value = "{clientId:.clientId}"
          }
  
          parameters {
            parameter_name  = "JsonParsingEngine"
            parameter_value = "JQ-1.6"
          }
        }
      }
    }
}

[esvm]

There is an open pull request related to this issue: #20769

[github-actions]

This functionality has been released in v3.66.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.