Terraform does not apply generated plan and clears out the state

[favoretti]

Terraform Version

Terraform v1.1.0
on darwin_amd64
+ provider registry.terraform.io/hashicorp/azuread v1.6.0
+ provider registry.terraform.io/hashicorp/azurerm v2.89.0
+ provider registry.terraform.io/hashicorp/helm v2.4.1
+ provider registry.terraform.io/hashicorp/kubernetes v1.13.4
+ provider registry.terraform.io/hashicorp/random v3.1.0
+ provider registry.terraform.io/hashicorp/tls v3.1.0

Terraform creates a plan, then doesn't even try to apply it and uploads an empty state..

Plan: 3 to add, 2 to change, 4 to destroy.

Changes to Outputs:
  ~ labels = (sensitive value)
╷
│ Warning: Experimental feature "module_variable_optional_attrs" is active
│
│   on .terraform/modules/aks/versions.tf line 3, in terraform:
│    3:     module_variable_optional_attrs,
│
│ Experimental features are subject to breaking changes in future minor or patch releases, based on feedback.
│
│ If you have feedback on the design of this feature, please open a GitHub issue to discuss it.
╵

Do you want to perform these actions?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes


Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Debug Output

Due to sensitivity of the log I can provide it upon request after some sanitizing.

Expected Behavior

Plan should have been applied.

Actual Behavior

Plan was not applied and an empty state (!) was uploaded back.

[jbardin]

Hi @favoretti,

Thanks for filing the issue, and sorry you are experiencing this. I think we are going to need some more information, especially the logs to figure out what might be going wrong here.

To reduce the amount of logs, and since the planning phase looks like it's working, you could save the plan to a file and only capture the logs from the apply step if you need. Plus if that changes the behavior at all, it could give us some more information for troubleshooting.

Is terraform being run completely locally or is this executing in Terraform Cloud, and how is your state being stored?

Thanks!

[favoretti]

Hi @jbardin,

Thanks, I completely understand the need for more information on this one.
To answer a few questions:

• Whether I run an apply with a pre-saved plan or just an apply with a refresh - doesn't matter, the behavior is exactly the same.
• Terraform is ran locally in this case, not terraform cloud.
• State backend is in an Azure Storage account. Same as our other 100 terraform repos, so this somehow is only specific to this particular use-case.

Now as for the log.. Can I share the trace log of apply phase with you in somewhat more secure fashion than posting it here?

Thanks,
Vlad

[jbardin]

Hi @favoretti,

You can encrypt the log file if you would like. If you are a keybase user you can encrypt the logs through there (same username for me, and GitHub is verified), or using gpg with the key at https://keybase.io/jbardin/pgp_keys.asc. If you are more comfortable using the Hashicorp's public key (https://www.hashicorp.com/security), we can also use that (just takes a little more time to get access to decrypt the content).

If you have any other ideas as what may be different for this particular configuration, it might also help with troubleshooting.

Thanks!

[favoretti]

Hi @jbardin,

I've uploaded trace log of apply to keybase. I've tried to sanitize the file as much as possible, while keeping what's important. Let me know if I over-sanitized it by accident :)

Thank you :)

[jbardin]

Thanks @favoretti, I think I have what I need. I haven't been able to reproduce this yet, but from manual inspection of the source I have a good hypothesis.

[favoretti]

Thanks, @jbardin!

[TonyLovesDevOps]

Glad to see this fixed; this happened to us but I couldn't nail down a solid repro so I didn't open an issue. Thankfully, we had versioning enabled on the bucket our state was stored in, so we were able to recover. That said, seeing no output back from terraform state list for a module that previously had around ~300 resources was definitely brown trousers time.

[github-actions]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.