dms-vpc-role is not configured properly when creating aws_dms_replication_instance

[matfreeman]

This is a similar (or the same) issue as terraform-providers/terraform-provider-aws#7748 which was closed.

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.12.16
+ provider.aws v2.39.0

Affected Resource(s)

• aws_dms_replication_subnet_group
• aws_dms_replication_instance

Terraform Configuration Files

# Roles defined as per official documentation:
# https://www.terraform.io/docs/providers/aws/r/dms_replication_instance.html

# Database Migration Service requires the below IAM Roles to be created before
# replication instances can be created. See the DMS Documentation for
# additional information: https://docs.aws.amazon.com/dms/latest/userguide/CHAP_Security.APIRole.html
#  * dms-vpc-role
#  * dms-cloudwatch-logs-role
#  * dms-access-for-endpoint

data "aws_iam_policy_document" "dms_assume_role" {
  statement {
    actions = ["sts:AssumeRole"]

    principals {
      identifiers = ["dms.amazonaws.com"]
      type        = "Service"
    }
  }
}

resource "aws_iam_role" "dms-access-for-endpoint" {
  assume_role_policy = "${data.aws_iam_policy_document.dms_assume_role.json}"
  name               = "dms-access-for-endpoint"
}

resource "aws_iam_role_policy_attachment" "dms-access-for-endpoint-AmazonDMSRedshiftS3Role" {
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonDMSRedshiftS3Role"
  role       = "${aws_iam_role.dms-access-for-endpoint.name}"
}

resource "aws_iam_role" "dms-cloudwatch-logs-role" {
  assume_role_policy = "${data.aws_iam_policy_document.dms_assume_role.json}"
  name               = "dms-cloudwatch-logs-role"
}

resource "aws_iam_role_policy_attachment" "dms-cloudwatch-logs-role-AmazonDMSCloudWatchLogsRole" {
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonDMSCloudWatchLogsRole"
  role       = "${aws_iam_role.dms-cloudwatch-logs-role.name}"
}

resource "aws_iam_role" "dms-vpc-role" {
  assume_role_policy = "${data.aws_iam_policy_document.dms_assume_role.json}"
  name               = "dms-vpc-role"
}

resource "aws_iam_role_policy_attachment" "dms-vpc-role-AmazonDMSVPCManagementRole" {
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonDMSVPCManagementRole"
  role       = "${aws_iam_role.dms-vpc-role.name}"
}

# Issue when creating aws_dms_replication_subnet_group 
# (required for aws_dms_replication_instance)
resource "aws_dms_replication_subnet_group" "replication_subnet" {
  replication_subnet_group_description = "Test replication subnet group"
  replication_subnet_group_id          = "test-dms-replication-subnet-group-tf"

  subnet_ids = "${aws_subnet.database_subnet.*.id}"

  # Explicit depends_on for required role
  depends_on = ["aws_iam_role.dms-vpc-role"]
}

Debug Output

Error applying plan:

• Error: AccessDeniedFault: The IAM Role arn:aws:iam::xxxxxxxx:role/dms-vpc-role is not configured properly. status code: 400, request id: xxxxxxxx
• on dms.tf line xxx, in resource "aws_dms_replication_subnet_group" "replication_subnet":
  xxx: resource "aws_dms_replication_subnet_group" "replication_subnet" {

Expected Behavior

On first terraform apply:

• • Apply complete! Resources: X added, 0 changed, 0 destroyed.

Actual Behavior

On first terraform apply:

Error applying plan:

• Error: AccessDeniedFault: The IAM Role arn:aws:iam::xxxxxxxx:role/dms-vpc-role is not configured properly. status code: 400, request id: xxxxxxxx
• on dms.tf line xxx, in resource "aws_dms_replication_subnet_group" "replication_subnet":
  xxx: resource "aws_dms_replication_subnet_group" "replication_subnet" {

On second terraform apply:

• Apply complete! Resources: X added, 0 changed, 0 destroyed.

Steps to Reproduce

1. terraform apply

[gblues]

I believe this is related to this issue:

#7600

In my experience, what solves both issues is to manage the DMS roles in a separate Terraform job.

In our case, we have a "top-level" terraform job that sets up our basic infrastructure and exports key objects, and then we have other jobs that leverage remote state to integrate with those exported objects.

When I moved the DMS role creation to the top-level job, both this issue and the above linked issue disappear:

• the DMS instance creation completes successfully the first time
• the ENI for the DMS instance successfully cleans up during tear-down

If I had to guess, I would say there is a missing dependency that both allows the DMS instance to start creating before the roles are fully provisioned, and allows the DMS roles to be deleted before the instance teardown has completed (which causes the ENI cleanup to fail).

I did try adding an explicit dependency on the DMS instance to the roles, which did not help.

[motilevy]

Just for additional info, running with -parallelism=1 also solves this issue.

[dave-irvine]

My workaround is to depend_on the attachment, rather than the role, and add a sleep. I think it takes the IAM change some time to propagate through so that DMS picks up that you have the permissions.

resource "aws_iam_role_policy_attachment" "dms-vpc-role-AmazonDMSVPCManagementRole" {
  policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonDMSVPCManagementRole"
  role       = aws_iam_role.dms-vpc.name

  # It takes some time for these attachments to work, and creating the aws_dms_replication_subnet_group fails if this attachment hasn't completed.
  provisioner "local-exec" {
    command = "sleep 30"
  }
}

resource "aws_dms_replication_subnet_group" "subnet-group" {
  replication_subnet_group_description = "Replication subnet group"
  replication_subnet_group_id          = "dms-replication-subnet-group"

  depends_on = [
    aws_iam_role_policy_attachment.dms-vpc-role-AmazonDMSVPCManagementRole
  ]
}

[divyenduz]

I can confirm that depends_on workaround does work, probably putting that in the docs is an option?

[aaronbrighton]

Just ran into this as well, depends_on (as stated in the documentation) is not adequate. The dirty sleep above seems to work for now. Until a cleaner fix can be implemented, a documentation update would be great!

[vadzimkaredzinkokoba]

I got the same error
but depends_on not working for me

[pp-assis]

depends_on + sleep worked for me ;)

[neil90]

depends_on + sleep worked for me as well

[leslie-alldridge]

+1 pls fix 🥳

[spicysomtam]

See this link. Might be worth following this advice on an aws account that has never used dms.

[giuseppeborgese]

By the way, I found the same issue also in CloudFormation.

[kvenugopal]

I tried both the options mentioned, however I am still getting this error. Not sure how to proceed. Appreciate your help and suggestions.

~ Tried depends On
~ Added the sleep timer

When I verify for the roles created in AWS console I see the required roles created with appropriate policy. Even on the second attempt to apply the change I still get the error.

Thanks
Karthik

[kvenugopal]

Hello All.
I could resolve this issue today.. Here is what I did

1. Manually created the "aws_iam_role" via one of the terraform. NOTE: the iam_role_name = "dms-vpc-role" ( hardcoded )

data "aws_iam_policy_document" "dms_assume_role" {
statement {
actions = ["sts:AssumeRole"]

principals {
  identifiers = ["dms.amazonaws.com"]
  type        = "Service"
}

}
}

resource "aws_iam_role" "dms-vpc-role" {
assume_role_policy = data.aws_iam_policy_document.dms_assume_role.json
name = "dms-vpc-role"
}

resource "aws_iam_role_policy_attachment" "dms-vpc-role-AmazonDMSVPCManagementRole" {
policy_arn = "arn:aws:iam::aws:policy/service-role/AmazonDMSVPCManagementRole"
role = aws_iam_role.dms-vpc-role.name
}

Once created made sure this role and policy "AmazonDMSVPCManagementRole" is attached to the role via AWS Console

2. Now I create the replication subnet group through different module

resource "aws_dms_replication_subnet_group" "subnet-group" {
replication_subnet_group_description = "Replication subnet group"
replication_subnet_group_id = "dms-replication-subnet-group"

depends_on = [
aws_iam_role_policy_attachment.dms-vpc-role-AmazonDMSVPCManagementRole
]
}

And it looks like this resource specifically looking for role named "dms-vpc-role" as defined in step 1.

It looks like there is a bug in the provider especially "resource "aws_dms_replication_subnet_group""

Thanks
Karthik

[zhelding]

A Retry step has been added to the create function for aws_replication_subnet_group.

This resolves the AccessDeniedFault error previously encountered when creating a repliction subnet group in the same operation as the necessary IAM policy-role attachments.

[github-actions]

This functionality has been released in v4.31.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

[ffelipek07]

Good Morning

My first post, here. I have the same problem, using the dms module, when I change the name "dms-vpc-role", to a custom name, I get the same error using the latest provider. Any idea ?

https://github.com/terraform-aws-modules/terraform-aws-dms/blob/v1.5.3/main.tf#L88

[kvenugopal]

@ffelipek07 ,
For whatever reason the resource "aws_dms_replication_subnet_group" is always looking for a IAM role named "dms-vpc-role" whose policy is ""arn:aws:iam::aws:policy/service-role/AmazonDMSVPCManagementRole"..

This IAM role name needs to be hardcoded.

Thanks

[ffelipek07]

I got your point, but i need change the name for exemple, dms-vpc-role-new. I receive this error.

[kvenugopal]

@ffelipek07
I guess there should be issue with this resource "aws_dms_replication_subnet_group" as it is looking for this specific IAM role. (dms-vpc-role).

Not sure how to go about it.

Thanks

[kvenugopal]

@ffelipek07 ,
May be you could try this.

1. Make sure in AWS account you don't have the role "dms-vpc-role" defined in IAM.
2. Go to DMS Server through AWS Console and create a DMS replication subnet group
3. Go back to IAM in AWS Account and look for dms-vpc-role.. If this exist which means AWS really wants to have this role created to setup DMS replication subnet group.

If AWS expects this role for whatever the reason.. then we have to live with it.

Thanks

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.