function(s) to merge a list (or map) of maps/objects to a single map/object

[tomalok]

Current Terraform Version

Terraform v0.12.13
(terragrunt version v0.21.2)

Use-cases

I have a dynamically-generated map of maps (or list of maps), and I'd like to merge them all down to a single map, using standard merge(map1, map2, ...) rules. FWIW, using merge() with objects also seems to work, as it's a shallow merge, so I'd like to keep that behavior, too.

I use these for picking up hierarchical variables from my directory structure (using terragrunt) -- if the same key is present when merging the maps, the one nearest to the local directory wins.

Attempted Solutions

# --- other inconsequential terragrunt stuff above here ---
locals {
  dir = get_terragrunt_dir()
  # map of yaml base filenames => whether we need to search parent folders
  var_yaml = {
    account = true
    region  = true
    swarm   = true
    local   = false
  }
  # resolve pathnames
  var_path = { for f, global in local.var_yaml:
    f => ( global
      ? "${local.dir}/${find_in_parent_folders("${f}.yaml", "NONE")}"
      : "${local.dir}/${f}.yaml"
    )
  }
  # load maps from paths
  vars = { for f, global in local.var_yaml:
    f => yamldecode(
      fileexists(local.var_path[f]) ? file(local.var_path[f]) : "{}"
    )
  }
  # TEST: dynamic merge?
  mk = keys(local.vars)
  m = [ for k in local.mk:
    merge( index(local.mk, k) == 0 ? {} : local.m[index(local.mk, k) - 1], local.vars[k] )
  ]
  # the last element of 'm' would be the complete merge (if this worked)
}

# this currently does what I'd like it to, but It's awkward/repetitive...
inputs = merge(
  local.vars["account"],
  local.vars["region"],
  local.vars["swarm"],
  local.vars["local"]
)

A plan results in:

[terragrunt] 2019/11/04 18:59:43 Not all locals could be evaluated:
[terragrunt] 2019/11/04 18:59:43 	- m
[terragrunt] 2019/11/04 18:59:43 Could not evaluate all locals in block.

Proposal

merge(...) accepts two or more maps as individual parameters, but does not accept a single parameter of map-of-maps or list-of-maps. If it's not possible to overload this function to allow for that, another function (or two) would take one or more maps-of-maps or lists-of-maps as parameters.

[teamterraform]

Hi @tomalok! Thanks for sharing this use-case.

The Terraform language has a general feature for turning lists/tuples into multiple arguments, by using the special symbol ... after the last argument expression, like this:

merge(maps...)

Because function arguments are ordered, this only works with sequence types (lists and tuples) and not for maps. However, if lexical sorting by key is sufficient for what you need -- which it sounds like should be true in your case where the different maps/objects have distinct keys/attributes -- you can use the values function to succinctly extract a list of the values from a map:

merge(values(local.vars)...)

Does that seems like it would meet the use-case you've shared here?

Since you're describing behaviors of Terragrunt's language rather than Terraform's language, we're not totally sure that the above would work there, but since Terragrunt is also using HCL as its foundation and this ... syntax is a HCL feature, hopefully it will work there too! If not, you'd need to open an issue in Terragrunt's own repository since we don't maintain Terragrunt here in this repository.

[Jaff]

I'm not able to extract a list of keys from a set of service configurations. I can get something 'resembling' a map:

lb_services = [
  {},
  {
    "key" = "service1"
  },
  {
    "key" = "service2"
  },
  {
    "key" = "service3"
  },
  {
    "key" = "service4"
  },
]

By applying this code:

locals {
  services_map = merge( var.services_1, var.services_2, var.services_3 )
  good_ports = flatten([
    for service in keys(local.services_map) : [
      for configs in local.services_map[service] : {
        key = configs.proxy_port > 0 ? service : null
      }
    ]
  ])
  lb_services = coalescelist(distinct(local.good_ports))
}

But if I try to get just the service name (key) from that list, I get:

Error: Invalid function argument

  on main.tf line 61, in locals:
  61:   sk = keys(tomap(local.lb_services))
    |----------------
    | local.lb_services is list of object with 5 elements

Invalid value for "v" parameter: cannot convert list of object to map of any
single type.

[tomalok]

@teamterraform thanks for the tip of ... operator to unroll lists into parameters, I'll check that out!

[tomalok]

@teamterraform - verified that ... does indeed do what I want, thanks!

locals {
  dir = get_terragrunt_dir()
  yaml_vars = [ "account", "region", "swarm", "local" ]
  # find and merge all specified yaml var files
  yaml_merged = merge(
    [ for p in
      [ for f in local.yaml_vars:
        fileexists("${local.dir}/${f}.yaml")
          ? "${local.dir}/${f}.yaml"
          : "${local.dir}/${find_in_parent_folders("${f}.yaml", "NONE")}"
      ]: yamldecode(fileexists(p) ? file(p) : "{}")
    ]...
  )
}

[teamterraform]

Great, thanks for confirming @tomalok!

In that case, we're going to close this out with the recommendation of using ... possibly in conjunction with values(...) as the primary way to address the use-case you described.

---

For future readers: note that the file we were discussing here was actually a Terragrunt configuration file rather than a Terraform configuration file, and thus not actually parsed by Terraform at all. It happens that the Terragrunt configuration file uses the same underlying expression language that Terraform does and so a Terraform answer was applicable to Terragrunt also in this case, but in general if you have feedback about the terragrunt.hcl file format then it would be better addressed to the terragrunt repository so that terragrunt's maintainers can see it and consider it.

[mltsy]

NOTE: This functionality is broken in some cases when trying to expand lists created using a for expression: #22404

[ghost]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.