Panic in hclwrite parser: didn't find any token of type TokenIdent

[apparentlymart]

Originally reported over in apparentlymart/terraform-clean-syntax#3 by @mattsawyer77.

On certain input, it seems that the analysis of traversals in expressions fails and produces a panic:

panic: didn't find any token of type TokenIdent

goroutine 1 [running]:
github.com/hashicorp/hcl/v2/hclwrite.inputTokens.PartitionType(0xc00036be20, 0x0, 0x0, 0xc0003c5fd8, 0x0, 0x0, 0x49, 0x0, 0x0, 0x0, ...)
        /Users/sawyer/gocode/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:88 +0x580
github.com/hashicorp/hcl/v2/hclwrite.inputTokens.PartitionTypeSingle(0xc00036be20, 0x0, 0x0, 0xc0003c5fd8, 0x0, 0x0, 0xc000000049, 0x0, 0x0, 0x0, ...)
        /Users/sawyer/gocode/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:92 +0xe3
github.com/hashicorp/hcl/v2/hclwrite.parseTraversalStep(0x130e460, 0xc000298050, 0xc00036be20, 0x0, 0x0, 0xc0003c5fd8, 0x0, 0x0, 0xc00036be20, 0x0, ...)
        /Users/sawyer/gocode/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:395 +0x364
github.com/hashicorp/hcl/v2/hclwrite.parseTraversal(0xc000263280, 0x3, 0x4, 0xc00036be20, 0x0, 0x0, 0xc0003c5fd8, 0x0, 0x0, 0xc00036be20, ...)
        /Users/sawyer/gocode/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:377 +0x3da
github.com/hashicorp/hcl/v2/hclwrite.parseExpression(0x130faa0, 0xc0000b6700, 0xc00036be20, 0x2, 0x2, 0xc0003c5fd8, 0x2, 0x2, 0x18b)
        /Users/sawyer/gocode/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:356 +0x175
github.com/hashicorp/hcl/v2/hclwrite.parseAttribute(0xc00028bea0, 0xc00036ba60, 0x0, 0x0, 0xc0003c5f88, 0x0, 0x0, 0xc00036b9a0, 0x0, 0x0, ...)
        /Users/sawyer/gocode/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:260 +0xaf4
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x130cc20, 0xc00028bea0, 0xc00036b9a0, 0x17, 0x17, 0xc0003c5f78, 0x17, 0x17, 0x0, 0x0, ...)
        /Users/sawyer/gocode/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:217 +0x632
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc0000b5a20, 0xc00036af20, 0x33, 0x33, 0xc0003c5e98, 0x33, 0x33, 0x0, 0x0, 0x0, ...)
        /Users/sawyer/gocode/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:193 +0x68d
github.com/hashicorp/hcl/v2/hclwrite.parseBlock(0xc000286b40, 0xc00036af20, 0x0, 0x0, 0xc0003c5e98, 0x0, 0x0, 0xc00036aa40, 0x5, 0x5, ...)
        /Users/sawyer/gocode/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:329 +0xf72
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x130cc60, 0xc000286b40, 0xc00036a9e0, 0x70, 0x70, 0xc0003c5e28, 0x70, 0x70, 0x0, 0x0, ...)
        /Users/sawyer/gocode/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:219 +0x4f7
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc0000b5b80, 0xc00033c000, 0x836, 0xa00, 0xc0003c2000, 0x836, 0x836, 0x0, 0x0, 0x0, ...)
        /Users/sawyer/gocode/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:193 +0x68d
github.com/hashicorp/hcl/v2/hclwrite.parse(0xc000102000, 0x38eb, 0x3aeb, 0xc0000e41e0, 0x1c, 0x1, 0x1, 0x0, 0xc0000c3c48, 0xc000000000, ...)
        /Users/sawyer/gocode/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:53 +0x2e4
github.com/hashicorp/hcl/v2/hclwrite.ParseConfig(...)
        /Users/sawyer/gocode/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/public.go:27
main.processFile(0xc0000e41e0, 0x1c, 0x1a4)
        /Users/sawyer/terraform-clean-syntax/main.go:75 +0xc8
main.processItem(0xc0000e41e0, 0x1c)
        /Users/sawyer/terraform-clean-syntax/main.go:52 +0x25d
main.processDir(0x7ffeefbff63f, 0x16)
        /Users/sawyer/terraform-clean-syntax/main.go:64 +0x119
main.processItem(0x7ffeefbff63f, 0x16)
        /Users/sawyer/terraform-clean-syntax/main.go:44 +0x17b
main.main()
        /Users/sawyer/terraform-clean-syntax/main.go:30 +0xca

It looks like the relevant bit of code is here-ish:

hcl/hclwrite/parser.go

Lines 391 to 400 in 7f9aa84

	case hcl.TraverseRoot, hcl.TraverseAttr:
	step := newTraverseName()
	children = step.inTree.children
	before, from, after = from.Partition(nativeStep.SourceRange())
	inBefore, token, inAfter := from.PartitionTypeSingle(hclsyntax.TokenIdent)
	name := newIdentifier(token)
	children.AppendUnstructuredTokens(inBefore.Tokens())
	step.name = children.Append(name)
	children.AppendUnstructuredTokens(inAfter.Tokens())
	return before, newNode(step), after

This codepath is assuming that any traversal step marked by the main parser as being a root (the beginning of a traversal) or an attribute (something that comes after a .) will contain an identifier in it. On the surface that seems like a reasonable assumption, but apparently there is at least one exceptional case where that isn't true.

[apparentlymart]

Thanks to @pdecat in apparentlymart/terraform-clean-syntax#2, we have the following small reproduction case:

a = google_pubsub_subscription.pubsub_subscriptions[count.index].name

[roni-frantchi]

Noticed #328 is merged was included in v2.3.0 - but I'm still getting this error for the following syntax:

records = ["${aws_elasticache_cluster.mgmt.cache_nodes.0.address}"]

[apparentlymart]

Hi @roni-frantchi,

Do you see the same crash if that's written in the non-legacy syntax aws_elasticache_cluster.mgmt.cache_nodes[0].address instead? I'm wondering if I missed an exception in the compatibility codepath that makes that still panic, in spite of the fix for the new syntax.

[roni-frantchi]

Thanks for the prompt reply @apparentlymart .
My crash when trying to parse a file with the aforementioned snippet looks like this:

panic: didn't find any token of type TokenOBrack

goroutine 1 [running]:
github.com/hashicorp/hcl/v2/hclwrite.inputTokens.PartitionType(0xc0003f1660, 0x2, 0x2, 0xc000400c88, 0x2, 0x2, 0xc00000005b, 0x0, 0x0, 0x0, ...)
        /Users/roni/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:88 +0x580
github.com/hashicorp/hcl/v2/hclwrite.parseTraversalStep(0x13359c0, 0xc0003e0cc0, 0xc0003f1660, 0x2, 0x2, 0xc000400c88, 0x2, 0x2, 0xc0003f1660, 0x0, ...)
        /Users/roni/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:408 +0xdab
github.com/hashicorp/hcl/v2/hclwrite.parseTraversal(0xc000289100, 0x5, 0x8, 0xc0003f1480, 0x9, 0x9, 0xc000400c60, 0x9, 0x9, 0xc0003f1360, ...)
        /Users/roni/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:377 +0x3da
github.com/hashicorp/hcl/v2/hclwrite.parseExpression(0x1337440, 0xc0003d21e0, 0xc0003f1360, 0xf, 0xf, 0xc000400c48, 0xf, 0xf, 0x40)
        /Users/roni/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:356 +0x175
github.com/hashicorp/hcl/v2/hclwrite.parseAttribute(0xc00047f7a0, 0xc0003f1360, 0x0, 0x0, 0xc000400c48, 0x0, 0x0, 0xc0003f1240, 0x0, 0x0, ...)
        /Users/roni/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:260 +0xaf4
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x1334580, 0xc00047f7a0, 0xc0003f1240, 0x14, 0x14, 0xc000400c30, 0x14, 0x14, 0x0, 0x0, ...)
        /Users/roni/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:217 +0x632
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc00027a580, 0xc0003f06a0, 0x33, 0x33, 0xc000400b38, 0x33, 0x33, 0x0, 0x0, 0x0, ...)
        /Users/roni/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:193 +0x68d
github.com/hashicorp/hcl/v2/hclwrite.parseBlock(0xc0002ad7a0, 0xc0003f06a0, 0x0, 0x0, 0xc000400b38, 0x0, 0x0, 0xc0003f0340, 0x0, 0x0, ...)
        /Users/roni/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:329 +0xf72
github.com/hashicorp/hcl/v2/hclwrite.parseBodyItem(0x13345c0, 0xc0002ad7a0, 0xc0003f0340, 0x3e, 0x3e, 0xc000400af0, 0x3e, 0x3e, 0x0, 0x0, ...)
        /Users/roni/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:219 +0x4f7
github.com/hashicorp/hcl/v2/hclwrite.parseBody(0xc00027a630, 0xc0003e8000, 0x19d, 0x200, 0xc000400000, 0x19d, 0x19d, 0x0, 0x0, 0x0, ...)
        /Users/roni/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:193 +0x68d
github.com/hashicorp/hcl/v2/hclwrite.parse(0xc000390900, 0x6a0, 0x8a0, 0xc00027f840, 0x3d, 0x1, 0x1, 0x0, 0xf, 0x0, ...)
        /Users/roni/go/pkg/mod/github.com/hashicorp/hcl/[email protected]/hclwrite/parser.go:53 +0x2e4
github.com/hashicorp/hcl/v2/hclwrite.ParseConfig(...)

Not quite the same error originally posted on this issue, but one I had thought was similar.

Making an edit and using aws_elasticache_cluster.mgmt.cache_nodes[0].address instead, works while parsing, but of course that syntax isn't valid in HCL1.

[apparentlymart]

Thanks @roni-frantchi! It sounds like there are still some similar cases like the ones I previously fixed, where the main HCL parser isn't producing accurate enough source location information.

I won't be able to prioritize further work on this right now because my focus is elsewhere, but I'll keep it on my list to take a look at next time I have some HCL cycles, if someone else doesn't get there before me.

[yaronya]

Hi @apparentlymart
Any update on this? 😺

[alisdair]

This should be fixed by #369.

[roni-frantchi]

Thanks @alisdair - confirmed it's working for us