terraform shows too much items in outputs when reducing the count of a create_before_destroy resource

[mildred]

I am opening this bug file to follow up #14040 which was mistakenly classified as a aws provider issue. But this is not the case as I could reproduce it using other resources. Instead, it seems to be tied to the create_before_destroy lifecycle option.

I could not reopen this last issue so I am creating this one instead. The aws provider issue hashicorp/terraform-provider-aws#725 ought to be closed too.

Terraform Version

Terraform v0.10.8

Terraform Configuration Files

variable "count" {
}

resource "local_file" "test" {
  count    = "${var.count}"
  filename = "tf-test-${count.index}"
  content  = "testing ${count.index}"

  lifecycle {
    create_before_destroy = true
  }
}

output "files" {
  value = "${formatlist("%v", local_file.test.*.filename)}"
}

Debug Output

https://gist.github.com/mildred/d767088784f4d2fe470edc4eea968ca4

Steps to Reproduce

terraform-0.10.8 init
terraform-0.10.8 apply -var count=4
terraform-0.10.8 apply -var count=2

Expected Behavior

The output from the last command (with -var count=2) should show two items.

Actual Behavior

Instead it shows:

files = [
    tf-test-0,
    tf-test-1,
    tf-test-2,
    tf-test-3
]

Important Factoids

This bug is also happening in terraform 0.9.3 and seems to be triggered by the create_before_destroy lifecycle property enabled. It can be reproduced using a aws_instance or aws_s3_bucket resource too.

References

• list output from aws_instance with count > 1 shows more items when reducing count #14040
• list output from aws_instance with count > 1 shows more items when reducing count terraform-provider-aws#725

[apparentlymart]

Hi @mildred! Sorry for this strange behavior.

It seems like what's going on here is that Terraform didn't update the output at all, for some reason. We made some changes in 0.11.0 to make outputs behave more robustly in the face of errors, so if you are able to try to reproduce this with Terraform 0.11.0 that could give more information about what's going on here.

[mildred]

Hi, thanks for replying.

The exact same test fails with terraform 0.11.0:

INS mildred@moiraine:dev/testbug16473$ vim terraform.tf
INS mildred@moiraine:dev/testbug16473$ ../bin/terraform-0.11.0 init
Initializing provider plugins...

Checking for available provider plugins on https://releases.hashicorp.com...
Downloading plugin for provider "local" (1.0.0)...

The following providers do not have any version constraints in configuration,

so the latest version was installed.
To prevent automatic upgrades to new major versions that may contain breaking

changes, it is recommended to add version = "..." constraints to the

corresponding provider blocks in configuration, with the constraint strings

suggested below.

provider.local: version = "~> 1.0"

Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see

any changes that are required for your infrastructure. All Terraform commands

should now work.
If you ever set or change modules or backend configuration for Terraform,

rerun this command to reinitialize your working directory. If you forget, other

commands will detect it and remind you to do so if necessary.

INS mildred@moiraine:dev/testbug16473$ ../bin/terraform-0.11.0 apply -var count=4
An execution plan has been generated and is shown below.

Resource actions are indicated with the following symbols:

create

Terraform will perform the following actions:


local_file.test[0]

id:       

content:  "testing 0"

filename: "tf-test-0"


local_file.test[1]

id:       

content:  "testing 1"

filename: "tf-test-1"


local_file.test[2]

id:       

content:  "testing 2"

filename: "tf-test-2"


local_file.test[3]

id:       

content:  "testing 3"

filename: "tf-test-3"


Plan: 4 to add, 0 to change, 0 to destroy.
Do you want to perform these actions?

Terraform will perform the actions described above.

Only 'yes' will be accepted to approve.
Enter a value: yes
local_file.test[0]: Creating...

content:  "" => "testing 0"

filename: "" => "tf-test-0"

local_file.test[3]: Creating...

content:  "" => "testing 3"

filename: "" => "tf-test-3"

local_file.test[2]: Creating...

content:  "" => "testing 2"

filename: "" => "tf-test-2"

local_file.test[1]: Creating...

content:  "" => "testing 1"

filename: "" => "tf-test-1"

local_file.test[3]: Creation complete after 0s (ID: 3780a8212b4fc6366ddef40288d92b53a2e81e1b)

local_file.test[0]: Creation complete after 0s (ID: 04035d9a988963b121a3e2794d43ee1c59d7fc2d)

local_file.test[1]: Creation complete after 0s (ID: 34c6f3a9765ad9a7cbd34aee6b94b0b0535c525c)

local_file.test[2]: Creation complete after 0s (ID: 4746934170add7c20d46cb0b0ce6cc12bc5cef10)
Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
Outputs:
files = [

tf-test-0,

tf-test-1,

tf-test-2,

tf-test-3

]

INS mildred@moiraine:dev/testbug16473$ ../bin/terraform-0.11.0 apply -var count=2

local_file.test[0]: Refreshing state... (ID: 04035d9a988963b121a3e2794d43ee1c59d7fc2d)

local_file.test[1]: Refreshing state... (ID: 34c6f3a9765ad9a7cbd34aee6b94b0b0535c525c)

local_file.test[3]: Refreshing state... (ID: 3780a8212b4fc6366ddef40288d92b53a2e81e1b)

local_file.test[2]: Refreshing state... (ID: 4746934170add7c20d46cb0b0ce6cc12bc5cef10)
An execution plan has been generated and is shown below.

Resource actions are indicated with the following symbols:

destroy

Terraform will perform the following actions:


local_file.test[2]


local_file.test[3]


Plan: 0 to add, 0 to change, 2 to destroy.
Do you want to perform these actions?

Terraform will perform the actions described above.

Only 'yes' will be accepted to approve.
Enter a value: yes
local_file.test[2]: Destroying... (ID: 4746934170add7c20d46cb0b0ce6cc12bc5cef10)

local_file.test[3]: Destroying... (ID: 3780a8212b4fc6366ddef40288d92b53a2e81e1b)

local_file.test[2]: Destruction complete after 0s

local_file.test[3]: Destruction complete after 0s
Apply complete! Resources: 0 added, 0 changed, 2 destroyed.
Outputs:
files = [

tf-test-0,

tf-test-1,

tf-test-2,

tf-test-3

]

INS mildred@moiraine:dev/testbug16473$ ../bin/terraform-0.11.0 version

Terraform v0.11.0

provider.local v1.0.0

[apparentlymart]

Thanks for the update, @mildred! The fact that it completed without an error on 0.11.0 indicates that this is a separate bug from what I expected, so we'll need to look into this more deeply to understand what is going on here.

[mildred]

Reading the code it seems as if the output node in the graph is evaluated too early when the second resource has not yet been deleted.

[mildred]

NodeApplyableOutput is considered applyable before the resource has been destroyed in terraform_output.go:47

[mildred]

In node_resource_apply.go, it clearly says

	// The "apply" side of a resource generally also depends on the
	// destruction of its dependencies as well. For example, if a LB
	// references a set of VMs with ${vm.foo.*.id}, then we must wait for
	// the destruction so we get the newly updated list of VMs.
	//
	// The exception here is CBD. When CBD is set, we don't do this since
	// it would create a cycle. By not creating a cycle, we require two
	// applies since the first apply the creation step will use the OLD
	// values (pre-destroy) and the second step will update.
	//
	// This is how Terraform behaved with "legacy" graphs (TF <= 0.7.x).
	// We mimic that behavior here now and can improve upon it in the future.
	//
	// This behavior is tested in graph_build_apply_test.go to test ordering.
	cbd := n.Config != nil && n.Config.Lifecycle.CreateBeforeDestroy
	if !cbd {
		// The "apply" side of a resource always depends on the destruction
		// of all its dependencies in addition to the creation.
		for _, v := range result {
			result = append(result, v+".destroy")
		}
	}

Another way to say it is that if create_before_destroy is set, we do not wait for destruction of resources before considering the dependent nodes to be applyable.

That's exactly why the dependent output node is considered applyable before the resources are destroyed.

[mildred]

Here clearly the outputs node is a leaf. It could wait for the resources to be destroyed without creating a cycle.

[apparentlymart]

Thanks for investigating that, @mildred!

create_before_destroy does indeed produce some tricky cases sometimes. We do have some special behaviors for outputs to try to mitigate this for those in particular but it seems like there is a case missed here.

(Outputs are somewhat "special" because evaluating them has no externally-visible side-effects, so it is safe to skip certain special behaviors for them in the interests of making sure they get updated correctly in more cases.)

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.