hashicorp/packer-plugin-sdk incompatible with zclconf/go-cty v1.11.0

[mschuchard]

Overview of the Issue

If the package github.com/zclconf/go-cty v1.11.0 is used with the Packer SDK, then an error is thrown. I am unsure if this is an error with Packer SDK or go-cty. If the error is with the other package, then please redirect me. I am opening this issue on both trackers.

Reproduction Steps

Do anything with a Packer plugin built with go-cty v1.11.0 as this occurs immediately after Packer loads the plugin.

Packer Plugin SDK/go-cty versions

packer-sdk 0.3.1 and 0.3.2 both throw this error.
go-cty v1.10.0 works perfectly. v1.11.0 throws this error.

Operating system and Environment details

go 1.18

Log Fragments and crash.log files

panic: ConfigSpec failed: gob: type cty.Type has no exported fields [recovered]
	panic: ConfigSpec failed: gob: type cty.Type has no exported fields

goroutine 1 [running]:
log.Panic({0xc000b3c2a8, 0xc000b3c2e8, 0xc000b3c318})
	/opt/hostedtoolcache/go/1.17.11/x64/src/log/log.go:354 +0x65
github.com/hashicorp/packer/packer.(*cmdProvisioner).checkExit(0x90, {0x43e1e60, 0xc00052daa0}, 0x0)
	/home/runner/work/packer/packer/packer/cmd_provisioner.go:47 +0x7f
github.com/hashicorp/packer/packer.(*cmdProvisioner).ConfigSpec.func1()
	/home/runner/work/packer/packer/packer/cmd_provisioner.go:19 +0x39
panic({0x43e1e60, 0xc00052daa0})
	/opt/hostedtoolcache/go/1.17.11/x64/src/runtime/panic.go:1038 +0x215
github.com/hashicorp/packer-plugin-sdk/rpc.(*commonClient).ConfigSpec(0xc0007905e0)
	/home/runner/go/pkg/mod/github.com/hashicorp/[email protected]/rpc/common.go:44 +0x297
github.com/hashicorp/packer/packer.(*cmdProvisioner).ConfigSpec(0xc000b3e0e0)
	/home/runner/work/packer/packer/packer/cmd_provisioner.go:22 +0x65
github.com/hashicorp/packer/hcl2template.decodeHCL2Spec({0x5ce8d10, 0xc000725e60}, 0xc00053e9a8, {0x7f7dd9ff6450, 0xc00053e9a8})
	/home/runner/work/packer/packer/hcl2template/decode.go:17 +0x39
github.com/hashicorp/packer/hcl2template.(*HCL2Provisioner).HCL2Prepare(0xc0005da5a0, 0xc0005da570)
	/home/runner/work/packer/packer/hcl2template/types.hcl_provisioner.go:51 +0x439
github.com/hashicorp/packer/hcl2template.(*PackerConfig).startProvisioner(0xc00056bb80, {{{0xc0005340f7, 0x6}, {0xc0005340fe, 0x6}}, {0x0, 0x0}, {0x5ce8d10, 0xc000724b10}}, 0xc0005526c0, ...)
	/home/runner/work/packer/packer/hcl2template/types.build.provisioners.go:199 +0x6a5
github.com/hashicorp/packer/hcl2template.(*PackerConfig).getCoreBuildProvisioner(0x0, {{{0xc0005340f7, 0x6}, {0xc0005340fe, 0x6}}, {0x0, 0x0}, {0x5ce8d10, 0xc000724b10}}, 0xc0005526c0, ...)
	/home/runner/work/packer/packer/hcl2template/types.packer_config.go:482 +0x105
github.com/hashicorp/packer/hcl2template.(*PackerConfig).getCoreBuildProvisioners(0xc0005340fe, {{{0xc0005340f7, 0x6}, {0xc0005340fe, 0x6}}, {0x0, 0x0}, {0x5ce8d10, 0xc000724b10}}, {0xc0005300e0, ...}, ...)
	/home/runner/work/packer/packer/hcl2template/types.packer_config.go:470 +0x1cb
github.com/hashicorp/packer/hcl2template.(*PackerConfig).GetBuilds(0xc00056bb80, {{0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, 0x0, 0x0, {0x0, ...}, ...})
	/home/runner/work/packer/packer/hcl2template/types.packer_config.go:670 +0x14fc
github.com/hashicorp/packer/command.(*BuildCommand).RunContext(0xc000c68ba0, {0x5ce83e0, 0xc000c6c980}, 0xc0007967e0)
	/home/runner/work/packer/packer/command/build.go:182 +0x21e
github.com/hashicorp/packer/command.(*BuildCommand).Run(0xc000c68ba0, {0xc00006e0a0, 0x2, 0x2})
	/home/runner/work/packer/packer/command/build.go:40 +0xc5
github.com/mitchellh/cli.(*CLI).Run(0xc0003bca00)
	/home/runner/go/pkg/mod/github.com/mitchellh/[email protected]/cli.go:262 +0x5f8
main.wrappedMain()
	/home/runner/work/packer/packer/main.go:262 +0xb28
main.realMain()
	/home/runner/work/packer/packer/main.go:49 +0xf3
main.main()
	/home/runner/work/packer/packer/main.go:35 +0x19

Additional Information

I suspect this is referencing the cty.Type from the <plugin>.hcl2spec.go. If I attempt to regenerate that file with go-cty v1.11.0, then the following error is thrown:

mapstructure-to-hcl2: main.[]: Unexpected package creation during export data loading

[apparentlymart]

Hi @mschuchard,

It seems like the Packer SDK is internally relying on the encoding/gob library for its wire serialization. Older versions of cty had some leftover support for gob which I added as a concession to HashiCorp Terraform before the Terraform team realized that gob wasn't going to work anyway, and so I recently removed that concession from cty because it was hampering ongoing maintenance.

It seems unfortunate that HashiCorp Packer was quietly relying on this behavior. I think the best thing to do here is first to see if there's some way that Packer can take responsibility for its own wire encoding concerns (that is: implement the gob encoding itself rather than relying on me to maintain it upstream.) If that isn't possible for some reason then I may consider adding back gob support, but if it comes to that then I'll likely have some design work to figure out how to reintroduce it in a way that doesn't make it so inconvenient to maintain other parts of this library.

I'll wait to see what the Packer team thinks before taking any action here.

[mschuchard]

I see now this overlaps with #132, which was also dual-reported on both trackers (which is how I discovered it). The chosen resolution was to downgrade to go-cty v1.10.0 for use with the packer-sdk.

I believe that this will probably not be acted upon in the packer-sdk, and therefore probably the path forward will be to lock to v1.10.0 for Packer moving forward. Hopefully that is not the case though.

[dbast]

Dependabot update proposals to v1.11.0 score with a compatibility of 3%, see e.g. mkaczanowski/packer-builder-arm#211 That maybe indicates lots of users relying on the packer-plugin-sdk compatibility.

[mschuchard]

I believe this is now being tracked officially at hashicorp/packer-plugin-sdk#187.