Add "explodeMap" function

[gmr]

explodeMap allows for maps created with scratch.MapSet to be treated the same way as KeyPair lists.

My use case is merging Consul KV pairs and Vault KV pairs together in my template, and then converting the scratch map to YAML/JSON/TOML in config files.

[catsby]

Hey @gmr - could you help me out and share some example usage? I'm still not 100% on consul-template so I'd like to see the input and expected output here, that should make code review much easier. Thanks!

[gmr]

Thanks, this would allow one to do something like:

{{ scratch.MapSet "example", "foo/bar", "a" }}
{{ scratch.MapSet "example", "foo/baz", "b" }}
{{ scratch.Get "example" | explodeMap | toYAML }}

Rendering would create:

foo:
  bar: a
  baz: b

The crux of the issue is that the explode which is used to create a structure that is usable by to toJSON, toTOML, toYAML, etc, takes pairs []*dep.KeyPair as the input parameter. The only way to create a datastructure with pairs []*dep.KeyPair is by way of querying consul.

The scratch data methods in consul template allow one to make a map data structure. explodeMap allows for one to create a scratch map, then spit it out to toml/json/yaml/etc.

My use case is to merge return values from consul.kv data and vault kv api v2 and normalize them into a single config map and then render them.

One alternative I can come up with that could let me get to the same conclusion would be a mapToKeyPairs type function, but that seemed a bit excessive given the extra attributes defined for a dep.KeyPair.

Another was to add scratch functions for creating an array of KeyPairs.

Ultimately this seemed like the simplest, most powerful way to get to allowing the user (who creates templates) to create nested data structures programmatically and then render them.

[gmr]

Looks like the build failed due to lack of deterministic sorting. I'm going to push a small test change that removes the sorting issue.

[gmr]

And finally an example of my use case as I think it would end up working:

{{ range tree "service/redis" }}{{ scratch.MapSet "config", .Key, .Value }}{{ end }}
{{ with secrets "kv2/service/redis" }}}{{ range $k, $v := .Data.data }}{{ scratch.MapSet "config", $k, $v }}{{ end }}{{ end }}
{{ scratch.Get "config" | explodeMap | toYAML }}

Conceptually this would take all KV pairs in the consul path and add them to the scratch map, and then all KV pairs in the Vault KV2 path add/overwrite them in the scratch map, then render the scratch map as YAML.

[gmr]

Is there any chance to get this included and a new release soon? I need to determine if we need to run our own forked copy or stay on the official release.

[pearkes]

@gmr depending on your timeline I'd recommend compiling this on your end -- we don't currently have a release scheduled though are trying to find resources for that early this year.

[eikenb]

Hey @gmr, since you submitted your pull request we've implemented a policy of requiring a signed CLA before merging. Would you please consider signing the CLA? Thanks.

[eikenb]

@gmr .. One last thing. I'm trying to get better test coverage of the functions going forward and while I see and appreciate the template test you include if you would add a simple test in funcs_test.go just to cover the function itself I'd greatly appreciate it.

I'm looking to make 0.22.0 a community PR merging release and would like to merge it at that time. Note I'll be doing the 0.21.1 bugfix release first.

Thanks.

[gmr]

Will do.

[eikenb]

@gmr Thanks.