AWS Route53 - Diffs didn't match during apply

[ghost]

On 0.4.0 and lower, when running terraform apply with Route 53 configurations we get the message 'diffs didn't match during apply'.

Here's the contents of the Route 53 config tf file:

resource "aws_route53_zone" "myapp" {
  name = "myapp.example.local"
}

resource "aws_route53_record" "gateway" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "gateway.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["${aws_instance.gateway.private_ip}"]
}

resource "aws_route53_record" "store" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "store.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["${aws_instance.store.private_ip}"]
}

resource "aws_route53_record" "keystore" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "keystore.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["${aws_instance.keystore.private_ip}"]
}

resource "aws_route53_record" "publisher" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "publisher.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["${aws_instance.publisher.private_ip}"]
}

resource "aws_route53_record" "analytics" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "analytics.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["${aws_instance.analytics.private_ip}"]
}

We were hoping this would be fixed by 0.4.0, but that does not appear to be the case as it's still happening on the latest release.

[catsby]

Thanks for reporting this @lyptt . Unfortunately I'm not able to reproduce this on either Terraform 0.4.0 that I just downloaded, or from the build of the current master branch (at 18f2e13).
I used the below pan, which is identical to yours except for use of private_ip references:

provider "aws" {
  region = "us-west-2"
}

resource "aws_route53_zone" "myapp" {
  name = "myapp.example.local"
}

resource "aws_route53_record" "gateway" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "gateway.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["127.0.0.2"]
}

resource "aws_route53_record" "store" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "store.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["127.0.0.3"]
}

resource "aws_route53_record" "keystore" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "keystore.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["127.0.0.5"]
}

resource "aws_route53_record" "publisher" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "publisher.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["127.0.0.6"]
}

resource "aws_route53_record" "analytics" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "analytics.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["127.0.0.7"]
}

I'm attempting another test with an AWS Instance and using it's private IP. Do you have any other information to help me reproduce this?

Thanks!

[catsby]

I modified the config to use the private IP of an instance and still can't reproduce the issue:

provider "aws" {
  region = "us-west-2"
}

resource "aws_route53_zone" "myapp" {
  name = "myapp.example.local"
}

resource "aws_instance" "nat" {
  ami = "ami-21f78e11"
  instance_type = "t1.micro"
  tags {
    Name = "testing-vpc-subnet-things"
  }
}

resource "aws_route53_record" "gateway" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "gateway.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["${aws_instance.nat.private_ip}"]
}

resource "aws_route53_record" "store" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "store.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["127.0.0.3"]
}

resource "aws_route53_record" "keystore" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "keystore.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["127.0.0.5"]
}

resource "aws_route53_record" "publisher" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "publisher.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["127.0.0.6"]
}

resource "aws_route53_record" "analytics" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "analytics.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["127.0.0.7"]
}

[ghost]

It looks like this is an issue when working with a tfstate file created in 0.3.7. Using tfstate files only created in 0.4.0 won't reproduce the issue.

This issue is prevalent in 0.3.7 and it appears that this issue then follows onto 0.4.0 via the tfstate file.

[ghost]

@catsby Another thing that seems to reproduce the issue, but I'm unsure how it's related, is when the target AMIs of the instances using these Route 53 records changes. I get exactly the same error when swapping out an old AMI with a new AMI.

We're having to work around this by running destroy, then pulling down the latest terraform config files and running apply on a blank slate, basically avoiding tfstate files entirely, which isn't ideal.

[ghost]

@catsby I've gotten down a decent way of reproducing this one. Based on the following config file:

provider "aws" {
  region = "us-west-2"
}

resource "aws_route53_zone" "myapp" {
  name = "myapp.example.local"
}

resource "aws_instance" "nat" {
  ami = "ami-21f78e11"
  instance_type = "t1.micro"
  tags {
    Name = "testing-vpc-subnet-things"
  }
}

resource "aws_route53_record" "gateway" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "gateway.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["${aws_instance.nat.private_ip}"]
}

resource "aws_route53_record" "store" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "store.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["127.0.0.3"]
}

resource "aws_route53_record" "keystore" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "keystore.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["127.0.0.5"]
}

resource "aws_route53_record" "publisher" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "publisher.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["127.0.0.6"]
}

resource "aws_route53_record" "analytics" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "analytics.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["127.0.0.7"]
}

Run an initial terraform apply with no tfstate files present. Then, update a property on your nat instance that'll cause it to be destroyed and recreated. Now run terraform apply again, and you'll be presented with the error:

Error applying plan:

1 error(s) occurred:

* 1 error(s) occurred:

* aws_route53_record.gateway: diffs didn't match during apply. This is a bug with Terraform and should be reported.

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

It seems that once an instance is destroyed, the Route 53 record is no longer able to be tracked, causing this bug.

I've tried merging in a fix for a similar issue #1430 into 0.4.0 (on the commit tagged 0.4.0), and this doesn't appear to fix the issue unfortunately. Any ideas?

[catsby]

I've tried merging in a fix for a similar issue #1430 into 0.4.0 (on the commit tagged 0.4.0), and this doesn't appear to fix the issue unfortunately. Any ideas?

Odd, I suspected #1430 to fix this; I can't reproduce on master :/

I used this config:

provider "aws" {
  region = "us-west-2"
}

resource "aws_route53_zone" "myapp" {
  name = "myapp.example.local"
}

resource "aws_instance" "nat" {
  ami = "ami-21f78e11"
  instance_type = "t1.micro"
  tags {
    Name = "testing-vpc-subnet-things"
  }
}

resource "aws_route53_record" "gateway" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "gateway.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["${aws_instance.nat.private_ip}"]
}

And changed it to this:

provider "aws" {
  region = "us-west-2"
}

resource "aws_route53_zone" "myapp" {
  name = "myapp.example.local"
}

resource "aws_instance" "nat" {
  ami = "ami-dfc39aef"
  instance_type = "t2.micro"
  tags {
    Name = "testing-vpc-subnet-things"
  }
}

resource "aws_route53_record" "gateway" {
  zone_id = "${aws_route53_zone.myapp.zone_id}"
  name = "gateway.myapp.example.local"
  type = "A"
  ttl = "300"
  records = ["${aws_instance.nat.private_ip}"]
}

The relevant plan bits:

-/+ aws_instance.nat
    ami:                      "ami-21f78e11" => "ami-dfc39aef" (forces new resource)
    instance_type:            "t1.micro" => "t2.micro" (forces new resource)
    private_dns:              "ip-172-31-17-1.us-west-2.compute.internal" => "<computed>"
    private_ip:               "172.31.17.1" => "<computed>"

~ aws_route53_record.gateway
    records.#: "" => "<computed>"

Everything worked as expected, the Route received the new record:

2015/04/10 09:23:14 [DEBUG] apply: aws_route53_record.gateway: executing Apply
aws_route53_record.gateway: Modifying...
  records.4086305246: "" => "172.31.39.85"

[...]

Apply complete! Resources: 1 added, 1 changed, 1 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

Can you check out the master branch, or even the recently released 0.4.1 and let me know? Thanks for all the help here.

[ghost]

This appears to be fixed in 0.4.2, thanks!

[catsby]

Great to hear, thanks for opening the issue!

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.