aws_dx_gateway_association_proposal recreated

[saliceti]

Terraform bug

It happens when associating a virtual private gateway with a direct connect gateway in a separate account (See AWS doc). There is a terraform resource for the proposal. But this resource is not permanent, so terraform will try to recreate it.

Terraform Version

• terraform 0.11.10
• terraform-provider-aws 2.12.0

Affected Resource(s)

• aws_dx_gateway_association_proposal
• aws_dx_gateway_association

Terraform Configuration Files

resource "aws_dx_gateway_association_proposal" "dcgw_association_proposal" {
  depends_on                  = ["aws_vpn_gateway_attachment.vpn_attachment"]
  dx_gateway_id               = "${local.dcgw_id}"
  dx_gateway_owner_account_id = "${local.shared_services_account_id}"
  associated_gateway_id       = "${aws_vpn_gateway.vpn_gw.id}"
}

resource "aws_dx_gateway_association" "dcgw_association" {
  provider = "aws.shared_services"

  proposal_id                         = "${aws_dx_gateway_association_proposal.dcgw_association_proposal.id}"
  dx_gateway_id                       = "${local.dcgw_id}"
  associated_gateway_owner_account_id = "${local.account_id}"

  timeouts {
    delete = "30m"
    create = "30m"
  }
}

Expected Behavior

• Apply: create proposal, associate in other account, networks are connected
• Any apply after that: No change

Actual Behavior

• Apply: create proposal, associate in other account
• Apply immediately: No change
• Apply after a few days once the proposal has been removed by AWS: create proposal, force recreation of association, networks are disconnected for up to 20min -> DOWNTIME

Steps to Reproduce

1. terraform apply
2. Wait for a few days or delete the proposal (Example)
3. terraform plan: shows creation of the proposal and recreation of the association

Important Factoids

Raised a support case with AWS to understand the behaviour. Here is the response:

I relayed the same question to the service team and they have informed me that the proposal will be available as long as the "accepter" has not accepted the proposal. Once the proposal is accepted, it will disappear after some time. There is no defined time duration as to how long the proposal is available after acceptance through CLI. As we use a distributed and eventual consistency model, I would highly recommend you to not use the time the proposal remains in the CLI output after acceptance to make any design, architectural, or application decisions.

[ewbankkit]

A solution here may be to modify the behavior of the aws_dx_gateway_association.proposal_id attribute so that on update if the new proposal was for the same Direct Connect gateway/associated gateway pair that no new association is created.
Maybe even allow the new value to be empty on update.
I think this could be achieved via a CustomizeDiffFunc.

[ewbankkit]

@saliceti How urgent is this to address? Thanks.

[saliceti]

We stopped using this resource and replaced it with boto code triggered by a provisionner.
It’s ugly but it works. So it’s not urgent anymore. Thanks.

[saliceti]

To be honest this resource is not useful in the current state. If no plan is made to fix it soon it should be removed in the meantime.

[josephschadlick]

I've just run into this problem as well. I tend to agree with @saliceti - this resource should be fixed or removed, as this bug is not obvious and has the potential to cause outages.

[acarrasquillo]

Just ran into this issue too. Terraform created a new proposal and detached the VPC from the Direct Connect Gateway.

[SureshGoli123]

I have also ran into this issue. It should be fixed as soon as possible. Anywork around for now?

[huddy]

We're also seeing this issue, we would prefer to avoid any messy work arounds obviously :)

[ewbankkit]

Looking at this some more the possibility I see is to change ForceNew to false for the proposal_id attribute of the aws_dx_gateway_association.
This would mean that once a DX Gateway Association has been created, any change to the proposal_id would not trigger a recreation of the resource.

[sporokh]

Same issue for me using the latest terraform version 0.12.24. Basically aws_dx_gateway_association_proposal removed by AWS itself within 1-3 days, so the actual id of the resource is no longer in AWS, which forces new creation.
Lifecycle ignore_changes is not helping either. The resource is literally unusable. Any estimates on the fix?

[hhh0505]

I'm experiencing the same issue with terraform version 0.12.24. We need a fix ASAP, this is causing downtime if we need to add or change any resoucres.

[piyat]

Encountered this too, going to omit until fixed.

[mattparkes]

Does anyone know if this has been fixed? It's still causing us a lot of grief.

[hhh0505]

Does anyone know if this has been fixed? It's still causing us a lot of grief.

Nope the PR is still open, I've been waiting months for this to get fixed

[gerl1ng]

I worked around the proposal recreation like this for now. It's not nice but it's at least usable.

variable "proposal_id" {
  default = ""
}

resource "aws_dx_gateway_association_proposal" "proposal" {
  count    = var.proposal_id == "" ? 1 : 0
  provider = aws.vgw

  dx_gateway_id               = var.dx_gateway_id
  dx_gateway_owner_account_id = data.aws_caller_identity.nethub.account_id
  associated_gateway_id       = aws_vpn_gateway.vgw.id
}

resource "aws_dx_gateway_association" "association" {
  provider = aws.nethub

  dx_gateway_id                       = var.dx_gateway_id
  associated_gateway_owner_account_id = data.aws_caller_identity.vgw.account_id

  # IMPORTANT: Set the proposal_id variable in the module call after the initial assocation was created.
  # Otherwise the connection will be recreated with a connectivity downtime of 30 minutes.
  proposal_id = var.proposal_id == "" ? aws_dx_gateway_association_proposal.proposal[0].id : var.proposal_id
  # Another safety-net implemented so the connectivity will not be lost by a recreation of the association. Overwrite to false if deletion is intendet
  lifecycle {
    prevent_destroy = true
  }
}

output "proposal_id" {
  value = var.proposal_id == "" ? aws_dx_gateway_association_proposal.proposal[0].id : var.proposal_id
}

After the initial association is created one needs to update the variable proposal_id to the output value. The proposal will not be recreated in the future and the association is not recreated either.

I did not test this yet with a new association, ony with associations that the proposal_id has been deleted by AWS.

[ghost]

This has been released in version 3.29.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template for triage. Thanks!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!