Terraform crashing on terraform show -json plan.cache

[davidalger]

Terraform Version

Terraform v1.0.0
on darwin_amd64
+ provider registry.terraform.io/hashicorp/archive v2.2.0
+ provider registry.terraform.io/hashicorp/google v3.68.0
+ provider registry.terraform.io/hashicorp/google-beta v3.68.0
+ provider registry.terraform.io/hashicorp/helm v2.1.2
+ provider registry.terraform.io/hashicorp/kubernetes v2.2.0
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/time v0.7.1

Terraform Configuration Files

...

Debug Output

$ terraform show plan.cache

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

$ terraform show -json plan.cache
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb8 pc=0x2d554dd]

goroutine 1 [running]:
github.com/hashicorp/terraform/internal/command/jsonplan.marshalPlanResources(0xc0026224e0, 0xc001afafa0, 0x1, 0x1, 0xc00072e950, 0xc0019b8180, 0xc002009900, 0xc0006c0000, 0x0, 0xc001c7ce10)
	/home/circleci/project/project/internal/command/jsonplan/values.go:161 +0xfd
github.com/hashicorp/terraform/internal/command/jsonplan.marshalPlanModules(0xc0026224e0, 0xc00072e950, 0xc001afe420, 0x3, 0x4, 0xc001c7d190, 0xc001c7d1c0, 0x0, 0x0, 0x0, ...)
	/home/circleci/project/project/internal/command/jsonplan/values.go:243 +0x1ba
github.com/hashicorp/terraform/internal/command/jsonplan.marshalPlanModules(0xc0026224e0, 0xc00072e950, 0xc001af9800, 0x13, 0x20, 0xc001c7d190, 0xc001c7d1c0, 0x49, 0x0, 0x0, ...)
	/home/circleci/project/project/internal/command/jsonplan/values.go:250 +0x47c
github.com/hashicorp/terraform/internal/command/jsonplan.marshalPlannedValues(0xc0026224e0, 0xc00072e950, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/home/circleci/project/project/internal/command/jsonplan/values.go:142 +0xaaa
github.com/hashicorp/terraform/internal/command/jsonplan.(*plan).marshalPlannedValues(0xc0011949a0, 0xc0026224e0, 0xc00072e950, 0x0, 0x0)
	/home/circleci/project/project/internal/command/jsonplan/plan.go:539 +0x70
github.com/hashicorp/terraform/internal/command/jsonplan.Marshal(0xc000515d40, 0xc001106140, 0xc00112c060, 0xc00072e950, 0x0, 0x0, 0x0, 0x0, 0x0)
	/home/circleci/project/project/internal/command/jsonplan/plan.go:127 +0x172
github.com/hashicorp/terraform/internal/command.(*ShowCommand).Run(0xc00052a000, 0xc000140070, 0x1, 0x2, 0xc00045a960)
	/home/circleci/project/project/internal/command/show.go:152 +0x966
github.com/mitchellh/cli.(*CLI).Run(0xc00051a280, 0xc000885200, 0x342bac3, 0x4)
	/home/circleci/go/pkg/mod/github.com/mitchellh/[email protected]/cli.go:262 +0x41a
main.wrappedMain(0x0)
	/home/circleci/project/project/main.go:384 +0x1112
main.realMain(0x0)
	/home/circleci/project/project/main.go:115 +0x554
main.main()
	/home/circleci/project/project/main.go:58 +0x25

Crash Output

https://gist.github.com/davidalger/2de8d19baf24901aaef670326cd856e1

Expected Behavior

Plans should be printed.

Actual Behavior

Terraform crashes.

Steps to Reproduce

1. terraform init
2. terraform plan -out plan.cache
3. terraform show -json plan.cache

Additional Context

Running terraform show plan.cache works while terraform show -json plan.cache crashes, due to the error seemingly being found buried in the JSON marshaling. This only occurs on 2 of dozens of Terraform pipelines (locally and in CI).

The code works with both 0.14.11 and 0.15.3, breaks in 0.15.4 and 0.15.5 as well as 1.0.0. The architecture doesn't matter, it's broken both on linux_amd64 and darwin_amd64

I can attempt to create an isolated reproduce case if need be, but may not be easy since it occurs on only two of our pipelines and I have no idea what triggers it.

References

[joe-a-t]

Just guessing, if the code works with all versions pre-0.15.4 then breaks on 0.15.4, it's probably because of the changes outside of terraform functionality that was introduced in https://github.com/hashicorp/terraform/releases/tag/v0.15.4 is part of the plan file (#28803 (comment)). This might be something where the -json wasn't updated. Also noticed this interesting FIXME in https://github.com/hashicorp/terraform/blob/main/internal/command/views/show.go#L18-L20.

[jbardin]

Hi @davidalger,

Thanks for filing the issue. The crash itself is relatively easy to fix, but I'm more interested in how you got to the state that managed to cause it. I don't have any ideas yet, other than I'm guessing it's related to deposed instances. Could you share the human-readable output from terraform show plan.cache in case it contains any clues?

[jbardin]

Ah, looking at it again, I see how you may have gotten here. It must be a deposed instance that no longer exists, resulting in a NoOp change, which is not accounted for in the json output logic.

[davidalger]

@jbardin Thanks for jumping on this. I've downloaded the Terraform binary artifact from Circle CI to test against the environments where this occurred, and the crash does in fact appear to be resolved on both environments where it was occurring. The plan is successfully rendered in JSON format, including a couple deposed no-op resources as you had figured were there.

As for how these no-op deposed resources got in there, I'm not 100% sure, but very likely related to some hiccups we had with GKE node pool creation not too long ago where the node pools got into a bizarre state from which the Google provider couldn't recover because it couldn't tear down the deposed node pools due to them being in some odd infinite auto-healing loop when they were created due to an issue GCP was having. After finally getting them deleted manually, which got Terraform unstuck, they created just fine the following morning…and apparently left behind the no-op deposed resources which are now (it seems) forever present in the state file since they have become no-ops.

Anyways, thanks again, and we look forward to 1.0.1 being released with the fix for this. Cheers!

[jbardin]

Thanks for the follow up @davidalger! The resources should get removed from the state, even though the end up being a no-op in the plan. Let me verify that however, as I've seen a few cases where deposed resources seem to be reported far more often than one would expect.

[rrijkse]

I think this issue should be re-opened this also happened on a project in our infrastructure that was upgrade to v0.15. Someone made a change directly on an S3 bucket.

# aws_s3_bucket.cloudtrail has been changed
  ~ resource "aws_s3_bucket" "cloudtrail" {
        id                          = "bucket-name"
        tags                        = {
            "Application" = "bucket"
            "CreatedBy"   = "Terraform"
            "Name"        = "name"
        }
        # (9 unchanged attributes hidden)

      ~ lifecycle_rule {
            id                                     = "trails"
          ~ prefix                                 = "*" -> "/*"
            tags                                   = {}
            # (2 unchanged attributes hidden)


            # (2 unchanged blocks hidden)
        }

[jbardin]

Thanks @rrijkse, but I'm not sure I understand. Are you saying that you can still replicate the above crash when testing with a build from the current v1.0 or main branch?

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.