DMS endpoint s3 target to support extra_connection_attributes

[karthik2web]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Description

DMS has introduced new attributes in s3_settings and api doc will be updated end of this week, it can be passed as either s3_settings or extra_connection_attributes.

Terraform for s3 target doesn't support extra_connection_attributes, it would be nice to add it so new features on endpoint can be utilized through extra_connection_attributes.

https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_dms_endpoint.go#L283

API which supports new attributes

aws dms create-endpoint --endpoint-identifier s3-target-endpoint --engine-name s3 --endpoint-type target
--s3-settings ‘{“ServiceAccessRoleArn”: “your-service-access-ARN”, “DataFormat”: “parquet”, "parquetVersion":"PARQUET_2_0"}’

aws dms create-endpoint --extra-connection-attributes "ServiceAccessRoleArn:your-service-access-ARN;bucketName=dms-test-321;bucketFolder=s3test;DataFormat=parquet;parquetVersion=PARQUET_2_0;"

New or Affected Resource(s)

• aws_dms_endpoint

Potential Terraform Configuration

resource "aws_dms_endpoint" "media_s3_endpoint" {
  endpoint_id  = ""
  endpoint_type = "target"
  extra_connection_attributes = "DataFormat=parquet;parquetVersion=PARQUET_2_0;"
  engine_name = "s3"
  s3_settings = {
    service_access_role_arn = ""
    bucket_name= ""
    bucket_folder = ""
    compression_type= "GZIP"
  }

  tags = ""

}

References

• #0000

[ewbankkit]

Related:

• S3 settings on aws_dms_endpoint conflict with "extra_connection_attributes" #8009

[ewbankkit]

@karthik2web The extra connection attributes for S3 and Parquet are listed here: dataFormat=parquet;parquetVersion=PARQUET_2_0.

[Masp333]

I'm facing the same issues and didn't find any workaround. Only after a lot of attempts to solve this i figured out that the problem was on the terraform side. It seems that the aws_dms_endpoint does not accept s3_settings and extra_connection_attributes at the same time.. Is there any possible solution to this, or you know when would it be possible to perform this with terraform?

[gabrielmoreira]

@Masp333 try this workaround, using local-exec provisioner:

locals {
  s3_settings = "compressionType=GZIP;dataFormat=parquet;parquetVersion=PARQUET_2_0;timestampColumnName=system_modified_date"
}

resource "aws_dms_endpoint" "endpoint" {
  endpoint_id                 = "${var.name}-target-endpoint-tf"
  endpoint_type               = "target"
  engine_name                 = "s3"
  extra_connection_attributes = "${local.s3_settings}"
  s3_settings {
    service_access_role_arn = "${var.project.endpoint_role_arn}"
    bucket_name      = "${aws_s3_bucket.s3.id}"
    bucket_folder    = "${var.folder}"
  }

  provisioner "local-exec" {
    command = "aws dms modify-endpoint --endpoint-arn '${aws_dms_endpoint.endpoint.endpoint_arn}' --s3-settings '${local.s3_settings}'"
  }
}

[Masp333]

@gabrielmoreira correct me if I'm wrong, but what I understood of your workaround you used a local-exec provisioner to modify the s3-settings right?

My problem is that I want to add the following attributes on the extra_connection_attributes:

addColumnName=true;
EncryptionMode=SSE_KMS;
ServerSideEncryptionKmsKeyId=arn:aws:kms:eu-west-1:xxxxxxxxx:key/xxxxxxxxxxxxxxxxxxxxxxxxx;

and for what I saw on documentation, S3_settings does not support addColumnName right? So, can I use this workaround to define this three attributes?
Thank you!

[imranismail]

This is sorely needed, we are still managing our DMS resources manually because of this. :(

[lyle-nel]

@imranismail Please upvote the pull request if you want it prioritised. It is not a full solution, but it will serve most of the use-cases out there by simply extending the supported s3_settings attributes to cover attributes that would otherwise need to be defined in extra_connection_attributes.

[k-scherer]

This is a huge probem, forcing us to manage DMS endpoints manually...

[rambej0201]

Hi everybody, on one of my data pipeline I am trying to implement time base partition for s3-target-endpoint as like we have TimeBasedPartitioner in io.confluent.connect.s3.S3SinkConnector

Do we have any related issue going on.

Thanks
Raghu

[noahsuresh]

Hi Hashicorp and other developers who are working on this issue can you please prioritize this and add this feature of adding an extra connection attribute to DMS endpoint as we gotta do an additional work of modifying it in console

[noahsuresh]

This is a huge probem, forcing us to manage DMS endpoints manually...

Agreed Yes

[noahsuresh]

@Masp333 try this workaround, using local-exec provisioner:

locals {
  s3_settings = "compressionType=GZIP;dataFormat=parquet;parquetVersion=PARQUET_2_0;timestampColumnName=system_modified_date"
}

resource "aws_dms_endpoint" "endpoint" {
  endpoint_id                 = "${var.name}-target-endpoint-tf"
  endpoint_type               = "target"
  engine_name                 = "s3"
  extra_connection_attributes = "${local.s3_settings}"
  s3_settings {
    service_access_role_arn = "${var.project.endpoint_role_arn}"
    bucket_name      = "${aws_s3_bucket.s3.id}"
    bucket_folder    = "${var.folder}"
  }

  provisioner "local-exec" {
    command = "aws dms modify-endpoint --endpoint-arn '${aws_dms_endpoint.endpoint.endpoint_arn}' --s3-settings '${local.s3_settings}'"
  }
}

Did you have a problem adding the s3 settings also back then in that terraform version whereas now we are able to add s3 setting

[ghost]

This has been released in version 3.24.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!