"Unrecognized remote plugin message" following 012upgrade instructions

[pmcevoy]

Terraform Version

*Please note the terraform12 alias I am using (terraform.exe is still v0.11)

 [master ≡]> terraform12 -version
Terraform v0.12.1
+ provider.aws v2.14.0

 [master ≡]> terraform --version
Terraform v0.11.14
+ provider.aws v2.14.0

(note - Windows 10)

I'm following the instructions on https://www.terraform.io/upgrade-guides/0-12.html#upgrading-to-terraform-0-12 to update from 0.11.14 to 0.12.1

Steps to Reproduce

I removed the .terraform directory from the CWD
Using terraform 11, I init and apply

. terraform init -backend-config="C:/Source/Infrastructure/Src/Environment/DingDev.tfbackend"
...
* provider.aws: version = "~> 2.14"
Terraform has been successfully initialized!

. terraform apply -var-file="C:/Source/Infrastructure/Src/Environment/DingDev.tfvars" -auto-approve
...
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

I then run 0.12checklist:

terraform 0.12checklist -var-file="C:/Source/Infrastructure/Src/Environment/DingDev.tfvars"
Looks good! We did not detect any problems that ought to be
addressed before upgrading to Terraform v0.12.

I have then tried a number of approaches, but I believe I need to init using version 12:

. terraform12 init -backend-config="C:/Source/Infrastructure/Src/Environment/DingDev.tfbackend"
...
* provider.aws: version = "~> 2.14"

I want to attempt the auto-upgrade, but it fails with seemling random provider errors that I am not using:

terraform12 0.12upgrade
...
-----------------------------------------------------------------------------

Error: failed to load provisioner "file": Unrecognized remote plugin message:

This usually means that the plugin is either invalid or simply
needs to be recompiled to support the latest protocol.

each time I run 0.12upgrade, it lists a different unreferenced provisioner: EG

Error: failed to load provisioner "habitat": Unrecognized remote plugin message:

Error: failed to load provisioner "local-exec": Unrecognized remote plugin message:

I'm pretty certain I'm following the upgrade instructions correctly, but it's failing.. I have a lot of configs using unsupported v 0.11 structures, and I'd like to fix as many automatically as I can

[pmcevoy]

Note, when I ran terraform12 init .. after the 11 init+apply, I did not see any incompatible construct warning/error messages. However when I try a terraform12 apply, I get a stack of warnings and errors for even a simple v0.11 config

[jbardin]

Hi @pmcevoy,

The plugins that are failing are all ones provided by the terraform binary itself. This leads me to believe that the main binary is locating and executing the old terraform binary for the internal plugins.

This seems to be windows specific at the moment. I'm not sure if it's a limitation of how windows processes cli arguments, or if we're not looking in the correct path. The complete log output may help show what is being picked for a binary internally. In the meantime I think you may need to consistently name the current version terraform.exe.

[pmcevoy]

I've completely removed all versions of terraform.exe from my path:

 [master ≡]> where.exe terraform
INFO: Could not find files for the given pattern(s).
PS v5 (0.1220114) C:\Source\Infrastructure\Src\Environment\Live\01-Networks\05-TransitGateway>
 [master ≡]> where.exe terraform12
INFO: Could not find files for the given pattern(s).

I setup the different versions of the exe in their own folders (not on my path) and ran the sequence above again from a clean configuration folder:

. C:\tools\terraform_0.11\terraform.exe init -backend-config="C:/Source/Infrastructure/Src/Environment/DingDev.tfbackend"
...
. C:\tools\terraform_0.11\terraform.exe apply -var-file="C:/Source/Infrastructure/Src/Environment/DingDev.tfvars" -auto-approve
...
. C:\tools\terraform_0.11\terraform.exe 0.12checklist -var-file="C:/Source/Infrastructure/Src/Environment/DingDev.tfvars"
Looks good! We did not detect any problems that ought to be

(that last message is odd because I know for certain the .tf config in my CWD uses a tags { construct)

. C:\tools\terraform_0.12\terraform.exe init -backend-config="C:/Source/Infrastructure/Src/Environment/DingDev.tfbackend"
...
. C:\tools\terraform_0.12\terraform.exe 0.12upgrade
..
-----------------------------------------------------------------------------
Error: failed to load provisioner "habitat": Unrecognized remote plugin message:

I've attached a DEBUG log of the entire set of above interactions. I hope you understand, I had to remove any AWS request/response logging I could find in the file - it's a pity that there isn't a feature to do this already)

Pete
Terraform.log

[mildwonkey]

🤔I think this is the important bit from that log:

2019-06-13T09:35:43.165+0100 [DEBUG] plugin.terraform.exe: 2019/06/13 09:35:43 [INFO] TF_CLI_ARGS value: "-no-color"
2019-06-13T09:35:43.165+0100 [DEBUG] plugin.terraform.exe: 2019/06/13 09:35:43 [INFO] CLI command args: []string{"internal-plugin", "-no-color", "provisioner", "habitat"}
2019-06-13T09:35:43.166+0100 [DEBUG] plugin.terraform.exe: 2019/06/13 09:35:43 Wrong number of args; expected: terraform internal-plugin pluginType pluginName

At a glance (without digging into the code) it looks like the extra TF_CLI_ARGS need to be left out when starting the internal plugin.

[apparentlymart]

That TF_CLI_ARGS bug is the same thing as #14847, I suspect.

---

Regarding the terraform 0.11checklist output: that command only produces warnings for the few things that must be changed before upgrading to Terraform 0.12. Changes like using block syntax for maps are handled after upgrading by the terraform 0.12upgrade tool. It's expected that most users will not see any warnings from terraform 0.11checklist, since most changes can be handled as part of upgrading rather than prior to upgrading.

[pmcevoy]

Ah - that fixed it. I think I had set no-color because the v11 colour scheme didn't play well with my Solarized-Dark powershell scheme and I couldn't see some text. The v12 scheme seems to play better with that so I'm happy to live without.

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.