Ability to iterate over object keys

[stan-sz]

Is your feature request related to a problem? Please describe.
I need to pass an object of objects as a parameter and iterate over keys and values to create resources (yes, blocked on #469). Example (modified from ARM docs):

param objectToTest object = {
  propA: {
    'propA-1': 'sub'
    'propA-2': 'sub'
  }
  propB: {
    'propB-1': 'sub'
    'propD-2': 'sub'
  }
  propC: {
    'propC-1': 'sub'
    'propC-2': 'sub'
  }
  propD: {
    'propD-1': 'sub'
    'propD-2': 'sub'
  }
}

output objectLength int = length(objectToTest)
output objectsALength int = length(objectToTest['propA'])
output objectsBLength int = length(objectToTest.propB)

Describe the solution you'd like

The last two lines of the example above demonstrate the ability to get the sub object through name index or as a named property (same for ARM JSON). The length function is key-aware as it properly returns 4 as the number of items at the top level of objectToTest (third line from the bottom), so dictionary awareness is already partially available. Two ideas come to my mind:

• let objects be int-indexable using copyIndex(); a syntax to retrieve the current key is also needed
• provide a function/property on object type to return the array of [{key1: value1}, ..., {keyN: valueN}] for the current level in the object

[stan-sz]

@alex-frankel has this been triaged? Can it be bound to a particular release?

[alex-frankel]

The only thing I know for sure is it won't make 0.4. We haven't prioritized anything beyond that, but we did get another ask for it this week, so I'd say it seems more likely than not for 0.5 or 0.6

cc @majastrz @shenglol - we were discussing this ask the other day.

[stan-sz]

Related to #1853

[bmoore-msft]

@stan-sz - can you expand on your use case for this one?

[stan-sz]

Our region configuration files are stored as dictionaries (resourceName: {properties of the resource}). Having the ability to iterate over dictionaries allows us to pass the dictionary as an input to a Bicep template without any intermediate step to pre-process the data so it can be consumed by today's Bicep.

[bmoore-msft]

@stan-sz - Can you double-click on that a bit? e.g. what's actually in the properties body and how you would iterate over it in bicep? Sounds like you're storing something as a dictionary when the platform models it as an array?

[stan-sz]

The config can be represented as:

<region name>:
  azureResources:
    storages:
      storage01:
        name: ...
        sku: ...
        ...
      storage02:
        name: ...
        sku: ...
        ...
    containerRegistries:
      acr01:
        name: ...
        sku: ...
        ...
      acr02:
        name: ...
        sku: ...
        ...

We are currently transforming these dicts into arrays and pass them to Bicep. The point of this issue is if we could just pass <region name>.azureResources.storages into parameters file and let Bicep iterate over the dictionary and create the expected accounts.

[bmoore-msft]

Understood - though that would also work as an array right? It sounds like you just have a different data model of a resources array. I'm trying to gauge whether this is preference (which is fine if it is) or if there's some broader scenario that need more thought overall...

[alex-frankel]

@stan-sz - I think the question we have is why is the input data shaped the way that it is? What system is it coming from that you are working with a dictionary and need to do the transform in the first place?

[stan-sz]

We use yaml template rendering engine that allows us to cross-link values in the file using the {{-notation. Having the data stored as dictionaries allows to reference some values from other resources. E.g.:

• have a resource group information in our yaml
• reference that resource group name in other resources

We'd like to transfer that relationship information to Bicep, but our yaml dictionaries are large and distributed across several git repos, so a conversion to array representation is out of scope. We do, though prepare the data as arrays in the ARM parameters files, but the point of this issue is to avoid exactly that.

Given that the length function is already dict-aware, we just need a way to iterate over (key, value) pairs at the given level of a dictionary.

[anthony-c-martin]

We had a discussion on this, and closed on the following:

items(<object_param>)

returns an array of objects with key & value properties set to the keys & values of the input object:
{
  key: <key>
  value: <value>
}

So for example, the following two are equivalent:

var result = items({
  key1: 'val1'
  key2: 'val2'
})

var result = [
  {
    key: 'key1'
    value: 'val1'
  }
  {
    key: 'key2'
    value: 'val2'
  }
]

Some example usages:

var myObj = {
  key1: 'val1'
  key2: 'val2'
}

// get the number of elements
var itemCount = length(items(myObj))

// get the keys
var keys = [for item in items(myObj): item.key]

// get the values
var values = [for item in items(myObj): item.value]

// using keys & values in a resource deployment
resource myRgs 'Microsoft.Resources/resourceGroups@2021-04-01' = [for item in items(myObj): {
  name: item.key
  location: 'West US'
  tags: {
    value: item.value
  }
}]

[anthony-c-martin]

I've implemented the above in the deployments engine, so it should be rolled out to production environments within the next couple of weeks. I'll update this thread when done, and we can then expose the function in Bicep.

[stan-sz]

I'm happy to hear about the progress and eagerly waiting for the news. Thanks a lot!

[slavizh]

nice.

[majastrz]

Btw, the array returned from the items() function will be sorted alphabetically by key.

[stan-sz]

@majastrz - can we have the function to retrieve the keys in the order they were in the original dictionary and, if the caller needs to have them sorted, provide the array-sorting function (#2251)?

[anthony-c-martin]

@majastrz - can we have the function to retrieve the keys in the order they were in the original dictionary and, if the caller needs to have them sorted, provide the array-sorting function (#2251)?

The ARM Deployment engine code unfortunately doesn't have any guarantees about preserving order, so we felt the safest option would be to order consistently (alphabetically), rather than have people take a dependency on a potentially false assumption about ordering.

[abatishchev]

Hey @anthony-c-martin, I can find this new items() function only in docs for Bicep. Is it available in ARM templates?

[slavizh]

@abatishchev https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/loops#dictionary-object

[abatishchev]

@slavizh yeah, it's documented here: https://docs.microsoft.com/en-us/azure/azure-resource-manager/bicep/bicep-functions-array#items
I wonder if it's available in ARM templates too, if no then how's it's implemented?

[bmoore-msft]

Run bicep build that should help...

[abatishchev]

@bmoore-msft here's what I've got

  "variables": {
    "copy": [
      {
        "name": "modifiedListOfEntities",
        "count": "[length(items(variables('entities')))]",
        "input": {
          "key": "[items(variables('entities'))[copyIndex('modifiedListOfEntities')].key]",
          "fullName": "[items(variables('entities'))[copyIndex('modifiedListOfEntities')].value.displayName]",
          "itemEnabled": "[items(variables('entities'))[copyIndex('modifiedListOfEntities')].value.enabled]"
        }
      }
    ],
    "entities": {
      "item002": {
        "enabled": false,
        "displayName": "Example item 2",
        "number": 200
      },
      "item001": {
        "enabled": true,
        "displayName": "Example item 1",
        "number": 300
      }
    }
  }

Does it mean that [items()] is an ARM template function too?

[alex-frankel]

Does it mean that [items()] is an ARM template function too?

Correct. If you see anything in the emitted ARM template from a bicep build, it is available in any ARM template. I can't remember about what we discussed about doc'ing the ARM template equivalent, but we probably should. cc @mumian / @stephaniezyen

[abatishchev]

ping @mumian / @stephaniezyen on missing docs.

[mumian]

@abatishchev - I will add the function to the ARM documentation.

[mumian]

fixed in https://github.com/MicrosoftDocs/azure-docs-pr/pull/194882.

[abatishchev]

@mumian I still don't see the function items() added to https://docs.microsoft.com/en-us/azure/azure-resource-manager/templates/template-functions

[mumian]

@abatishchev - sorry, I will work on it next Monday.

[mumian]

@abatishchev - the items() function has been added to the article.

[abatishchev]

@mumian awesome, thanks!

[dperezro-ins]

Excude me, but this items() function work with object of objects? i.e.

param objectToTest object = {
  propA: {
    'propA-1': {
      'propA-1-1': 'sub'
      'propA-1-2': 'sub'
    }
    'propA-2': {
      'propA-2-1': 'sub'
      'propA-2-2': 'sub'
    }
  }
  propB: {
    'propB-1': {
      'propB-1-1': 'sub'
      'propB-1-2': 'sub'
    }
    'propB-2': {
      'propB-2-1': 'sub'
      'propB-2-2': 'sub'
    }
  }
  propC: {
    'propC-1': {
      'propC-1-1': 'sub'
      'propC-1-2': 'sub'
    }
    'propC-2': {
      'propC-2-1': 'sub'
      'propC-2-2': 'sub'
    }
  }
  propD: {
    'propD-1': {
      'propD-1-1': 'sub'
      'propD-1-2': 'sub'
    }
    'propD-2': {
      'propD-2-1': 'sub'
      'propD-2-2': 'sub'
    }
  }
}

// expected output
output listOfObjects array = items(objectToTest)

/*
listOfObjects = [
  {
    key: propA
    value: {
      'propA-1': {
        'propA-1-1': 'sub'
        'propA-1-2': 'sub'
      }
      'propA-2': {
        'propA-2-1': 'sub'
        'propA-2-2': 'sub'
      }
    }
  }
  {
    key: propB
    value: {
      'propB-1': {
        'propB-1-1': 'sub'
        'propB-1-2': 'sub'
      }
      'propB-2': {
        'propB-2-1': 'sub'
        'propB-2-2': 'sub'
      }
    }
  }
......
]
*/

[majastrz]

Yes. The items() function iterates over the first level of properties and doesn't care what the values of those properties are.

[RiverHeart]

var myObj = {
  key1: 'val1'
  key2: 'val2'
}

// get the number of elements
var itemCount = length(items(myObj))

// get the keys
var keys = [for item in items(myObj): item.key]

// get the values
var values = [for item in items(myObj): item.value]

// using keys & values in a resource deployment
resource myRgs 'Microsoft.Resources/resourceGroups@2021-04-01' = [for item in items(myObj): {
  name: item.key
  location: 'West US'
  tags: {
    value: item.value
  }
}]

This is pretty good but I wish you could pass in the key name like items(myObj, "name") so I can pass the result into properties that only accept NameValuePairs and not KeyValuePairs. Appsettings for Sites are like this and while I can iterate on them to construct a NVP object it feels needlessly complex.

[stan-sz]

@RiverHeart consider opening a new issue and reference this one for visibility and triage.