Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

panic in pulumi convert from terraform - gcp cloud-foundation-fabric #27

Closed
giterinhub opened this issue Sep 6, 2023 · 8 comments
Closed

panic in pulumi convert from terraform - gcp cloud-foundation-fabric #27

giterinhub opened this issue Sep 6, 2023 · 8 comments
Assignees
@Frassle
Labels
impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec p1 Bugs severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed
Milestone
0.94

Comments

@giterinhub
Copy link

giterinhub commented Sep 6, 2023 •
edited by justinvp
Loading

What happened?

Cloned Terraform modules from https://github.com/GoogleCloudPlatform/cloud-foundation-fabric.git

Tried to convert fast/stages/0-bootstrap from Terraform to Pulumi Typescript.

On Windows, error:

cloud-foundation-fabric\fast\stages\0-bootstrap>pulumi convert --from terraform --language typescript --out pulumi
Converting from terraform...
Downloading provider: terraform
Downloading provider: gcp
Downloading provider: local
panic: fatal: A failure has occurred: each.value seen during expression conversion, but each scope not set
goroutine 36 [running]:
github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/go/common/util/contract/failfast.go:23
github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Failf({0x2903ef6?, 0xc0005ef920?}, {0x0?, 0x4?, 0x269b160?})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/go/common/util/contract/fail.go:32 +0xcd
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.rewriteTraversal(0x200?, 0xc00037ba80, {0x40?, 0x10d?}, {0xc00047a4c0?, 0x3, 0x4})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:1193 +0x238e
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.convertScopeTraversalExpr(...)
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:1246
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.convertExpression(0x2da1701?, 0x0?, 0xc000ada500?, {0xc000048540?, 0x5e?}, {0x2da1a18?, 0xc00091d440?})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:1471 +0x12e
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.convertBody(0xc0010495b0, 0xc00037ba80, {0xc000380a50, 0x2f}, {0x2da17e8, 0xc00054ca50})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:1722 +0x1955
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.convertBody(0xc0010495b0, 0xc00037ba80, {0xc000380900, 0x28}, {0x2da17e8, 0xc00054cb00})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:1687 +0x10d3
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.convertBody(0xc0010495b0, 0xc00037ba80, {0xc0002f2320, 0x1f}, {0x2da17e8, 0xc000a40000})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:1659 +0x2265
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.convertManagedResources(0xc0010495b0, {0x2d88400, 0xc000528570}, 0xc00037ba80, 0xc000cdc000, 0xc000108870)
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:2106 +0x7dd
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.translateModuleSourceCode(0x1d?, {0x2dad1e0, 0x3e7de48}, {0xc000ada480, 0x38}, {0x2dad160, 0xc000374400}, {0xc0008f03e0, 0x19}, {0x2d88400, ...})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:2825 +0x3825
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.translateModuleSourceCode(0x48?, {0x2dad1e0, 0x3e7de48}, {0xc0002de640, 0x37}, {0x2dad160, 0xc000374400}, {0x2d78430, 0x1}, {0x2d88400, ...})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:2497 +0x6ea6
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.TranslateModule(...)
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:2917
main.(*tfConverter).ConvertProgram(0x30, {0xc000459800?, 0xc000b0b980?}, 0xc000b0b9b0)
        /Users/runner/work/pulumi-converter-terraform/pulumi-converter-terraform/cmd/pulumi-converter-terraform/main.go:57 +0x24b
github.com/pulumi/pulumi/sdk/v3/go/common/resource/plugin.(*converterServer).ConvertProgram(0xc0008dcae0, {0x2da0e10, 0xc000b0b980}, 0xc000548000)     /Users/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/go/common/resource/plugin/converter_server.go:65 +0xf7
github.com/pulumi/pulumi/sdk/v3/proto/go._Converter_ConvertProgram_Handler.func1({0x2da0e10, 0xc000b0b980}, {0x271db20?, 0xc000548000})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/converter_grpc.pb.go:122 +0x78
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x2da0e10, 0xc000b0ad20}, {0x271db20, 0xc000548000}, 0xc000ac6120, 0xc000ab6180)
        /Users/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x3e8
github.com/pulumi/pulumi/sdk/v3/proto/go._Converter_ConvertProgram_Handler({0x25c2600?, 0xc0008dcae0}, {0x2da0e10, 0xc000b0ad20}, 0xc00020e070, 0xc0008dc960)
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/converter_grpc.pb.go:124 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc00080a3c0, {0x2daad00, 0xc000585040}, 0xc000352b40, 0xc0008d6a50, 0x3de54b8, 0x0)
        /Users/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1337 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc00080a3c0, {0x2daad00, 0xc000585040}, 0xc000352b40, 0x0)
        /Users/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1714 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        /Users/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:959 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /Users/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:957 +0x18c
error: rpc error: code = Unavailable desc = error reading from server: read tcp 127.0.0.1:64283->127.0.0.1:64282: wsarecv: An existing connection was forcibly closed by the remote host.

On Linux, Error:

~/cloud-foundation-fabric/fast/stages/0-bootstrap$ pulumi convert --from terraform --generate-only --language typescript
Converting from terraform...
panic: fatal: A failure has occurred: each.value seen during expression conversion, but each scope not set
goroutine 21 [running]:
github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.failfast(...)
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/go/common/util/contract/failfast.go:23
github.com/pulumi/pulumi/sdk/v3/go/common/util/contract.Failf({0x21c590b?, 0xc000b5f980?}, {0x0?, 0x4?, 0x1f61de0?})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/go/common/util/contract/fail.go:32 +0xcd
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.rewriteTraversal(0x200?, 0xc0003b4d00, {0x43?, 0x10d?}, {0xc000447980?, 0x3, 0x4})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:1193 +0x238e
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.convertScopeTraversalExpr(...)
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:1246
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.convertExpression(0x2655401?, 0xe0?, 0xc0003255e0?, {0xc00030e240?, 0x5c?}, {0x2655698?, 0xc000e4cc60?})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:1471 +0x12e
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.convertBody(0xc000fd75b0, 0xc0003b4d00, {0xc000444600, 0x2f}, {0x2655468, 0xc00046efd0})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:1722 +0x1955
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.convertBody(0xc000fd75b0, 0xc0003b4d00, {0xc0004445a0, 0x28}, {0x2655468, 0xc00046f080})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:1687 +0x10d3
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.convertBody(0xc000fd75b0, 0xc0003b4d00, {0xc0001d8360, 0x1f}, {0x2655468, 0xc000d774a0})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:1659 +0x2265
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.convertManagedResources(0xc000fd75b0, {0x263c0c0, 0xc0002ca9d0}, 0xc0003b4d00, 0xc000a0f680, 0xc00046a420)
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:2106 +0x7dd
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.translateModuleSourceCode(0x1d?, {0x2660d60, 0x36ef698}, {0xc000059900, 0x3b}, {0x2660ce0, 0xc0002ea400}, {0xc0002f44a0, 0x19}, {0x263c0c0, ...})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:2825 +0x3825
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.translateModuleSourceCode(0x48?, {0x2660d60, 0x36ef698}, {0xc00050e240, 0x3a}, {0x2660ce0, 0xc0002ea400}, {0x262c3b8, 0x1}, {0x263c0c0, ...})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:2497 +0x6ea6
github.com/pulumi/pulumi-terraform-bridge/v3/pkg/tf2pulumi/convert.TranslateModule(...)
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi-terraform-bridge/[email protected]/pkg/tf2pulumi/convert/tf.go:2917
main.(*tfConverter).ConvertProgram(0x30, {0xc000088400?, 0xc0002be600?}, 0xc0002be630)
        /Users/runner/work/pulumi-converter-terraform/pulumi-converter-terraform/cmd/pulumi-converter-terraform/main.go:57 +0x24b
github.com/pulumi/pulumi/sdk/v3/go/common/resource/plugin.(*converterServer).ConvertProgram(0xc000a21940, {0x2654a90, 0xc0002be600}, 0xc0000a60c0)       /Users/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/go/common/resource/plugin/converter_server.go:65 +0xf7
github.com/pulumi/pulumi/sdk/v3/proto/go._Converter_ConvertProgram_Handler.func1({0x2654a90, 0xc0002be600}, {0x1fe3720?, 0xc0000a60c0})
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/converter_grpc.pb.go:122 +0x78
github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1({0x2654a90, 0xc0002be2a0}, {0x1fe3720, 0xc0000a60c0}, 0xc0002ea0a0, 0xc0002d41e0)
        /Users/runner/go/pkg/mod/github.com/grpc-ecosystem/[email protected]/go/otgrpc/server.go:57 +0x3e8
github.com/pulumi/pulumi/sdk/v3/proto/go._Converter_ConvertProgram_Handler({0x1e8a940?, 0xc000a21940}, {0x2654a90, 0xc0002be2a0}, 0xc00085e0e0, 0xc000a21840)
        /Users/runner/go/pkg/mod/github.com/pulumi/pulumi/sdk/[email protected]/proto/go/converter_grpc.pb.go:124 +0x138
google.golang.org/grpc.(*Server).processUnaryRPC(0xc000978780, {0x265e880, 0xc0004e6340}, 0xc0002dc000, 0xc000b5ee70, 0x367d2d8, 0x0)
        /Users/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1337 +0xdf3
google.golang.org/grpc.(*Server).handleStream(0xc000978780, {0x265e880, 0xc0004e6340}, 0xc0002dc000, 0x0)
        /Users/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:1714 +0xa36
google.golang.org/grpc.(*Server).serveStreams.func1.1()
        /Users/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:959 +0x98
created by google.golang.org/grpc.(*Server).serveStreams.func1
        /Users/runner/go/pkg/mod/google.golang.org/[email protected]/server.go:957 +0x18c
error: rpc error: code = Unavailable desc = error reading from server: EOF

Expected Behavior

Successful conversion from Terraform to Pulumi Typescript

Steps to reproduce

  1. git clone https://github.com/GoogleCloudPlatform/cloud-foundation-fabric.git
  2. cd cloud-foundation-fabric/fast/stages/0-bootstrap
  3. pulumi convert --from terraform --language typescript

Output of pulumi about

Unix
`CLI
Version 3.80.0
Go Version go1.21.0
Go Compiler gc

Host
OS ubuntu
Version 22.04
Arch x86_64

Pulumi locates its logs in /tmp by default`

Windows
`CLI
Version 3.80.0
Go Version go1.21.0
Go Compiler gc

Host
OS Microsoft Windows 11 Pro
Version 10.0.22621 Build 22621
Arch x86_64

Backend
Name pulumi.com`

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).
@giterinhub giterinhub added kind/bug Some behavior is incorrect or out of spec needs-triage Needs attention from the triage team labels Sep 6, 2023
@Frassle Frassle transferred this issue from pulumi/pulumi Sep 6, 2023
@Frassle
Copy link
Member

Frassle commented Sep 6, 2023

This is a bug in the converter plugin itself, not pulumi codegen. Moving to relevant repo.

@Frassle Frassle removed the needs-triage Needs attention from the triage team label Sep 6, 2023
@justinvp justinvp added impact/panic This bug represents a panic or unexpected crash p1 Bugs severe enough to be the next item assigned to an engineer labels Sep 7, 2023
@justinvp justinvp added this to the 0.94 milestone Sep 7, 2023
@justinvp justinvp transferred this issue from pulumi/pulumi-terraform-bridge Sep 7, 2023
@Frassle
Copy link
Member

Frassle commented Sep 8, 2023

OK so it's not ideal this is a panic, but it's panic because the terraform code itself is actually malformed.

That repo has some dynamic blocks that refer to an "each" variable which doesn't actually exist: https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/blob/master/modules/bigquery-dataset/main.tf#L88-L97 (added by PR GoogleCloudPlatform/cloud-foundation-fabric#1480).

So we have a decision to make about what we should do in this case.

Currently we panic.
We could instead just return an error to the user pointing out what's broken in their TF code.
We could abuse the notImplemented function to return the expression but in a way that won't work.
We could just return an invalid expression.

Open to thoughts on what would be most user friendly here.

@aureq
Copy link
Member

aureq commented Sep 8, 2023

I'll probably lean toward user education and pointing out what's broken in their TF code. One they get a chance to address the issue since they own their own code, and two, we don't hide/mask something is incorrect in the first place.

Happy to hear what others think about.

@justinvp
Copy link
Member

justinvp commented Sep 8, 2023

We could instead just return an error to the user pointing out what's broken in their TF code.

I'd lean towards this. Is this easy enough to do for this case and other cases of invalid TF code? Or will it end up being a case-by-case thing converting various features from TF to PCL?

I see how the other options could be attractive, if we wanted the conversion to "succeed," but then the user is going to have to go fix-up the converted pulumi code that won't work. Seems better to tell them up-front that the TF is invalid (if we can) and needs to be fixed before it can be converted.

@Frassle
Copy link
Member

Frassle commented Sep 8, 2023

Is this easy enough to do for this case and other cases of invalid TF code? Or will it end up being a case-by-case thing converting various features from TF to PCL?

It's not hard, but a bit of grunt work because errors currently aren't fed through this part of the code.

It will end up being a case-by-case thing. There's a few places in the converter that still panic and I haven't looked into them all deeply to see if it's an assumption that should always hold, or only hold for valid Terraform.

@giterinhub
Copy link
Author

I'd like to be able to debug the error, to know where the problem in Terraform lies. How can I do that?
I'd then rather fix the terraform code. Thanks for all the inputs.

@Frassle
Copy link
Member

Frassle commented Sep 8, 2023

I'll fix this up to return an error pointing to the TF source location, but in the meantime for this case the error is the dynamic blocks in https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/blob/master/modules/bigquery-dataset/main.tf#L88-L97 which are using "each" as an iteration variable. They should be using the dynamic block name instead.

@Frassle Frassle added the resolution/fixed This issue was fixed label Sep 11, 2023
@Frassle
Copy link
Member

Frassle commented Sep 11, 2023

I've update the converter to error not panic on this. Converting the linked code above now gives a set of errors:

error: /home/fraser/projects/tfexamples/cloud-foundation-fabric/modules/bigquery-dataset/main.tf:93,22-43: each.value cannot be used in this context; A reference to "each.value" has been used in a context in which it is unavailable, such as when the configuration no longer contains the value in its "for_each" expression. Remove this reference to each.value in your configuration to work around this error.
error: /home/fraser/projects/tfexamples/cloud-foundation-fabric/modules/bigquery-dataset/main.tf:94,22-41: each.value cannot be used in this context; A reference to "each.value" has been used in a context in which it is unavailable, such as when the configuration no longer contains the value in its "for_each" expression. Remove this reference to each.value in your configuration to work around this error.
error: /home/fraser/projects/tfexamples/cloud-foundation-fabric/modules/bigquery-dataset/main.tf:92,22-43: each.value cannot be used in this context; A reference to "each.value" has been used in a context in which it is unavailable, such as when the configuration no longer contains the value in its "for_each" expression. Remove this reference to each.value in your configuration to work around this error.
error: /home/fraser/projects/tfexamples/cloud-foundation-fabric/modules/bigquery-dataset/main.tf:104,24-45: each.value cannot be used in this context; A reference to "each.value" has been used in a context in which it is unavailable, such as when the configuration no longer contains the value in its "for_each" expression. Remove this reference to each.value in your configuration to work around this error.
error: /home/fraser/projects/tfexamples/cloud-foundation-fabric/modules/bigquery-dataset/main.tf:105,24-45: each.value cannot be used in this context; A reference to "each.value" has been used in a context in which it is unavailable, such as when the configuration no longer contains the value in its "for_each" expression. Remove this reference to each.value in your configuration to work around this error.
error: /home/fraser/projects/tfexamples/cloud-foundation-fabric/modules/bigquery-dataset/main.tf:116,22-43: each.value cannot be used in this context; A reference to "each.value" has been used in a context in which it is unavailable, such as when the configuration no longer contains the value in its "for_each" expression. Remove this reference to each.value in your configuration to work around this error.
error: /home/fraser/projects/tfexamples/cloud-foundation-fabric/modules/bigquery-dataset/main.tf:117,22-43: each.value cannot be used in this context; A reference to "each.value" has been used in a context in which it is unavailable, such as when the configuration no longer contains the value in its "for_each" expression. Remove this reference to each.value in your configuration to work around this error.
error: /home/fraser/projects/tfexamples/cloud-foundation-fabric/modules/bigquery-dataset/main.tf:118,22-43: each.value cannot be used in this context; A reference to "each.value" has been used in a context in which it is unavailable, such as when the configuration no longer contains the value in its "for_each" expression. Remove this reference to each.value in your configuration to work around this error.

I'll close this and get a new release of the converter published.

@Frassle Frassle closed this as completed Sep 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Frassle Frassle

Labels
impact/panic This bug represents a panic or unexpected crash kind/bug Some behavior is incorrect or out of spec p1 Bugs severe enough to be the next item assigned to an engineer resolution/fixed This issue was fixed
Projects
None yet
Milestone
0.94
Development

No branches or pull requests
4 participants
@justinvp @aureq @Frassle @giterinhub