Resource aws_dms_endpoint - Conflict between extra_connection_attributes and s3_settings

[MauriceBrg]

Terraform CLI and Terraform AWS Provider Version

$ terraform -v
Terraform v0.15.5
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v3.44.0

Affected Resource(s)

• aws_dms_endpoint

Terraform Configuration Files

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~>3.0"
    }
  }
}

provider "aws" {
  region = "eu-central-1"
}

resource "aws_s3_bucket" "target_bucket" {
  bucket_prefix = "reproducible-bucket"
  force_destroy = true
}

resource "aws_iam_role" "dms_s3_writer" {
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [
      {
        Action = "sts:AssumeRole"
        Effect = "Allow"
        Sid    = ""
        Principal = {
          Service = "dms.amazonaws.com"
        }
      }
    ]
  })
}

resource "aws_dms_endpoint" "endpoint" {
  endpoint_type = "target"
  engine_name   = "s3"
  endpoint_id   = "dummy-endpoint"

  extra_connection_attributes = join(";", [
    "parquetTimestampInMillisecond=true",
    "CdcPath=cdc",
    "dataFormat=parquet",
    "PreserveTransactions=true",
    "includeOpForFullLoad=true",
    "timestampColumnName=timestamp",
    "parquetVersion=PARQUET_2_0",
    "bucketName=${aws_s3_bucket.target_bucket.id}",
    "bucketPrefix=some/prefix",
    "datePartitionEnabled=true"
  ])

  s3_settings {
    service_access_role_arn = aws_iam_role.dms_s3_writer.arn
  }
}

Debug Output

Note: I changed my Account ID and IAM Username, aside from that it's the plain output.

https://gist.github.com/MauriceBrg/9ea8a732379413d2c72b9028de439656

Panic Output

Expected Behavior

A terraform plan after a successful apply shouldn't find any differences.

Actual Behavior

The terraform plan after the apply wants to change the configuration of the DMS-S3-Endpoint in a way that would remove the bucket name and other vital configuration properties.

My guess:

There are two places in the resource to configure certain properties: s3_settings and extra_connection_attributes.
Some configurations can be made in both places, the problem seems to arise when you configure settings in the extra_connection_attributes that could be configured under s3_settings. In that case the s3_settings seem to be preferred and overwrite the extra_connection_attributes. Interestingly only on updates, the initial creation works just fine.

Steps to Reproduce

1. terraform apply -auto-approve
2. terraform plan
3. The code didn't change, but it wants to remove the bucket name and change other settings.

Important Factoids

None.

References

• feat(dms_endpoint): add DataFormat, EncryptionMode, and KmsKeyId to the S3 settings block #14340
• Add dms endpoint s3 attributes to keep format from changing to default on update #14620
• resource/dms_endpoint: add support for extraConnectionAttributes in create/read ops and handle diffs #16827
• Add S3 Connection Attributes to DMS Endpoint #17591
• r/aws_dms_endpoint: diff in extra_connection_attributes present when configured with s3_settings #19501
• Database Migration Service provider 'resource_aws_dms_endpoint.go' breaks/drops extra-connection-attributes due to conditional logic error #20290
• aws_dms_endpoint - Add support for DMS Endpoint settings #23225

[anGie44]

Likely related #19501

[YakDriver]

@MauriceBrg Thanks for raising this! This is an interesting looking diff problem. Have you had a chance to try v4.2.0? The s3_settings were altered a lot in #20913 that was part of v4.2.0. If you have a chance, let us know if that helps or not.

[YakDriver]

@MauriceBrg After looking more carefully at your configuration, here are some points.

extra_connection_attributes

As you have it, your configuration will not work well. AWS is reportedly going to deprecate extra_connection_attributes at some point also. The best approach is to entirely drop extra_connection_attributes and use s3_settings only. See below for an example.

preserveTransactions

preserveTransactions=true is not compatible with data_format=parquet. You will either need to change to csv or not preserve transactions.

Example Config Using s3_settings

resource "aws_dms_endpoint" "test" {
  endpoint_type = "target"
  engine_name   = "s3"
  endpoint_id   = "dummy-endpoint"

  s3_settings {
    bucket_folder                    = "some/prefix"
    bucket_name                      = aws_s3_bucket.test.id
    cdc_path                         = "cdc"
    data_format                      = "parquet"
    date_partition_enabled           = true   
    include_op_for_full_load         = true
    parquet_timestamp_in_millisecond = true
    parquet_version                  = "parquet-2-0"
    service_access_role_arn          = aws_iam_role.test.arn
    timestamp_column_name            = "timestamp"
  }
}

[YakDriver]

This issue should be resolved in v4.59.0 by using aws_dms_s3_endpoint (and an additional issue #28748 with aws_dms_replication_task in v4.50.0). Please let us know if not.

[github-actions]

This functionality has been released in v4.50.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.