Resources in nested template not shown if reference is used in the parameter values

[J0F3]

Describe the bug
Probably related to #83 but not sure so I though it might worth it to post it here.

To Reproduce
An ARM Template with a nested deployment. The parameter value of the nested deployment uses the reference function.
When deploying the template with Whatif it shows only the vNet created in the parent template and nothing about the Subnet created in the nested template.
On a re-deployment it gets even more weird. Whatif shows that the subnet gets deleted which is actually not what appends. So very confusing.

When the parameter value is replaced by a literal string the Whatif output is shown the changes correctly. So assume it is also related to the use of the reference function.

ARM Template:

{
  "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "functions": [],
  "resources": [
    {
      "type": "Microsoft.Resources/deployments",
      "apiVersion": "2019-10-01",
      "name": "vnetmodule",
      "properties": {
        "expressionEvaluationOptions": {
          "scope": "inner"
        },
        "mode": "Incremental",
        "parameters": {
          "location": {
            "value": "[resourceGroup().location]"
          }
        },
        "template": {
          "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
          "contentVersion": "1.0.0.0",
          "parameters": {
            "location": {
              "type": "string"
            }
          },
          "functions": [],
          "resources": [
            {
              "type": "Microsoft.Network/virtualNetworks",
              "apiVersion": "2020-05-01",
              "name": "vNet1",
              "location": "[parameters('location')]",
              "properties": {
                "addressSpace": {
                  "addressPrefixes": [
                    "10.0.0.0/16"
                  ]
                }
              }
            }
          ],
          "outputs": {
            "resourceName": {
              "type": "string",
              "value": "vNet1"
            },
            "resourceId": {
              "type": "string",
              "value": "[resourceId('Microsoft.Network/virtualNetworks', 'vNet1')]"
            }
          }
        }
      }
    },
    {
      "type": "Microsoft.Resources/deployments",
      "apiVersion": "2019-10-01",
      "name": "subnetModule",
      "properties": {
        "expressionEvaluationOptions": {
          "scope": "inner"
        },
        "mode": "Incremental",
        "parameters": {
          "vNetName": {
            "value": "[reference(extensionResourceId(resourceGroup().id, 'Microsoft.Resources/deployments', 'vnetmodule'), '2019-10-01').outputs.resourceName.value]"
          }
        },
        "template": {
          "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
          "contentVersion": "1.0.0.0",
          "parameters": {
            "vNetName": {
              "type": "string"
            }
          },
          "functions": [],
          "resources": [
            {
              "type": "Microsoft.Network/virtualNetworks/subnets",
              "apiVersion": "2020-05-01",
              "name": "[format('{0}/subnet1', parameters('vNetName'))]",
              "properties": {
                "addressPrefix": "10.0.0.0/24"
              }
            }
          ]
        }
      },
      "dependsOn": [
        "[extensionResourceId(resourceGroup().id, 'Microsoft.Resources/deployments', 'vnetmodule')]"
      ]
    }
  ]
}

Expected behavior
I would except that the vnet and subnet is shown as 'create' on the first deployment and as 'no change' in subsequent deployment.

Screenshots
First deployment (with reference in parameter value). Subnet is missing in the output:

Re-deployment (with the same template). Subnet is shown as 'delete':

First deployment when the value of the 'vNetName' parameter is stet to a literal string. Both the vnet and the subnet is shown correctly in the output:

Re-deployment (with the same template) show both resource correctly (beside except some noise probably not related to this issue):

Client
PowerShell

Additional context
I noticed quite some of such strange outputs of Whatif during playing with Biceps and Biceps modules as this results in templates with nested deployment where often also the reference function is used. So I thought it will be worth to place it here. But maybe your are already very aver of that? 😊

[alex-frankel]

Basically, what-if short circuits too aggressively when we see a reference() function, but we should still be able to calculate all of the resource IDs required for what-if without evaluating the reference() call. So this should be doable.

Thanks for reporting!

[JustinGrote]

@alex-frankel I ran into this one when using modules in bicep. Anything in a module (which translates to a nested deployment) doesn't show up in my whatif result.

[JFolberth]

One comment to expand on running with cli 2.2 and bicep the plan actually ignores the child resources rather than delete like the original issue described:

[alex-frankel]

We discussed this today and the root cause of the issue is essentially that as soon as there is one parameter that uses a runtime function in a nested template, we have no good way of ignoring it/throwing it away and evaluating the rest of the template. The reason it is difficult for us because the way we evaluate expressions in an ARM Template is...fragile :) We have some work ongoing to refactor this code to make this easier for us, but no good ETA I can provide.

@J0F3 - for your case in particular, you can probably refactor the code to avoid the reference() call entirely. You can set the vnet name to be a variable in the parent template, then pass that variable to the nested template as a parameter.

[dani3lheidemann]

Hi @alex-frankel,
I recently opened #3237 and can confirm that it's the same issue.

I also can confirm that when I'm leaving out any references, that validation shows up all resources I actually want to deploy. And of course, I'm able to set additional variables to "convert" my input parameters, but I also have to waive using ARM/Bicep module outputs as input parameters for other modules, because using outputs is also defined as a reference in ARM.

May you have an update when this is gonna be fixed?

Thank you very much!

[alex-frankel]

We are working on this now with the hopes of having it resolved by the end of the calendar year. This is a bit of a gnarly, low level issue so please be patient :)

[alex-frankel]

FYI - the fix for this has been checked in. Should be deployed sometime in December.

[samhughes91]

Very happy to hear this news. Thanks for the update Alex

[J0F3]

Awesome news! Thank you @alex-frankel!

[alex-frankel]

Yep, I generally agree with this sentiment. The plan workflow is reliable in those tools.

Things will be much better with this issue solved, but we need to be able to fix the what-if noise issue too to make what-if truly reliable, yeah?

[flcdrg]

True, Terraform does have 'plan', but at best it's a cross-your-fingers-maybe and at worst a complete lie - so many times I've had 'plan' succeed, only for 'apply' to fail. It sounds like what-if might be similar, though ideally it would be better (than the low bar I've experienced with terraform plan at least!)

[marcindulak]

True, Terraform does have 'plan', but at best it's a cross-your-fingers-maybe and at worst a complete lie - so many times I've had 'plan' succeed, only for 'apply' to fail. It sounds like what-if might be similar, though ideally it would be better (than the low bar I've experienced with terraform plan at least!)

Terraform plan and apply are not fully reliable hashicorp/terraform-provider-azurerm#25585.
Some of the problems are due to Azure API changes, and some due to flakiness of the cloud.

[JustinGrote]

True, Terraform does have 'plan', but at best it's a cross-your-fingers-maybe and at worst a complete lie - so many times I've had 'plan' succeed, only for 'apply' to fail. It sounds like what-if might be similar, though ideally it would be better (than the low bar I've experienced with terraform plan at least!)

Terraform plan and apply are not fully reliable hashicorp/terraform-provider-azurerm#25585. Some of the problems are due to Azure API changes, and some due to flakiness of the cloud.

There's a difference here. I'm not expecting a guaranteed outcome of a whatif plan. What I expect is effectively a definitive list of every API call and property change it is going to make. There's no guarantee there won't be a conflict or a transient change that causes that to fail, but at least I have an idea of the steps being taken.

When whole swaths of stuff are missing, that's when it's a problem, and suddenly things are deleted that I wasn't told it was going to attempt to delete.

[theonlyway]

True, Terraform does have 'plan', but at best it's a cross-your-fingers-maybe and at worst a complete lie - so many times I've had 'plan' succeed, only for 'apply' to fail. It sounds like what-if might be similar, though ideally it would be better (than the low bar I've experienced with terraform plan at least!)

Terraform plan and apply are not fully reliable hashicorp/terraform-provider-azurerm#25585. Some of the problems are due to Azure API changes, and some due to flakiness of the cloud.

The terraform plan is merely telling you what it's "going" to do and what it thinks the outcome will be (If it isn't waiting on a projected value that is only known after other resources have executed in the graph). It's not a guarantee it is going to be successful and that's not what people are after (If you wanted that you would have a rather heavy linting/language server validating every single one of your inputs). Its output, however, is enough to make a decision in regard to whether or not you want to actually execute it. For me it's always executed what the plan said it's going do. Whereas I've had bicep execute things the what-if said it wasn't going to do.
When Terraform apply fails after the plan it's 99% related to the values being passed to Azure API and nothing, not even ARM knows it's going to fail until it executes because you still get that behavior in Bicep.

I've never had to second guess a terraform plan whereas I have had the Bicep what-if be so wrong multiple times that I literally can't trust it for anything and once you stop trusting a tool it's hard to get that back.

[flavienbwk]

Bicep's leadership needs to prioritize this work. Bicep advantages are clear (that's why a lot of disccusions happen there), but critical components such as what-if need to be addressed in order to make Bicep a production-ready tool.

[alex-frankel]

@flavienbwk et al - can everyone help confirm that I am not missing something here?

The work is fully prioritized, which is why we are hoping to have this particular issue resolved in early October. From the responses I am seeing here, it sounds like as long as we are not outright missing resources the core problem with what-if is solved, even though the noise problem will NOT be resolved. Is my understanding correct?

Please keep in mind that there are ways to declare resource names that lead to them not being resolvable at deploy-time. We block this in most cases, but you can work around it with modules. With the release of fixing this short-circuiting issue, we are also going to ship a new linter that warns you about this behavior (which we consider an anti-pattern). Just something to keep in mind to set expectations properly.

[tjementum]

can everyone help confirm that I am not missing something here?

I guess we will have to wait and see. As described in a previous post, I found a workaround to actually have --what-if work.

BUT there is a big problem even if what-if shows a plan. Resources that are “calculated” will show up as being potentially changed on every single deployment. See the following screenshot of a deployment that has NO changes:

Because all my environment variables are “calculated,” they show up with the current value in orange and the formula in green. I know the green formular will result in the same value as the orange, so nothing will be changed. But it pollutes the output and makes it next to impossible to see what actual changes will be done. If I was using Terraform and a state file, Terraform would show what changes I did.

Have you solved this, so you able to “suppress” a formular that will result in the same output?

Also, the screenshot shows that, e.g., properties.logAnalyticsDestinationType: "AzureDiagnostics" will be deleted. BUT the fact is it will not. This is because ARM does things behind the scenes.

So, I fear that when you “fix” this issue, you have an equally big task to get ALL the different Azure ARM Providers updated to behave correctly. But since this issue has been hiding this problem, the different teams do not even know that there is a problem.

[cathei]

Can't believe this major issue is open for 3+ years and no warnings are built in, people had to search to get here. I did found working workaround for me after several attempt. Where myConnectionString is mod1's computed output and we want to pipe it to mod2:

module mod1 'mod1.bicep' = {
	name: 'mod1'
}

module mod2 'mod2.bicep' = {
	name: 'mod2'
	params: {
		myConnectionString: mod1.outputs.?myConnectionString ?? '(known after created)'
	}
}

Combining safe-dereference .? and coalesce ?? default value like above template worked for me. At least what-if doesn't hide all the resources to be created with mod2. After deploying the template, the value was set properly on actual resource.

Hope this helps.

[alex-frankel]

Have you solved this, so you able to “suppress” a formular that will result in the same output?

Also, the screenshot shows that, e.g., properties.logAnalyticsDestinationType: "AzureDiagnostics" will be deleted. BUT the fact is it will not. This is because ARM does things behind the scenes.

So, I fear that when you “fix” this issue, you have an equally big task to get ALL the different Azure ARM Providers updated to behave correctly. But since this issue has been hiding this problem, the different teams do not even know that there is a problem.

@tjementum this are examples of the noise problem I was describing and resolving this issue (#157) is not going to solve that problem. We are currently investigating options for making meaningful progress on the noise issue that does not require RP/service team involvement.

cc @azcloudfarmer / @dantedallag / @snarkywolverine

[Ramu3l]

Hi @alex-frankel ,
since we are so close to October 1, are you confident we can release this feature on time?
Thank you for your support!

[daviddavis]

Hi @alex-frankel. It's October. Was just wondering if you had any updates to share. Thanks.

[azcloudfarmer]

Hello Everyone - we are excited to share the deployment of this fix is in progress and we will begin onboarding tenants gradually as soon as the rollout is complete. For those of you interested on getting your tenant onboarded to be able to receive this fix sooner, please fill this survey. We thank you all for the patience! 📎

[jikuja]

Hello Everyone - we are excited to share the deployment of this fix is in progress and we will begin onboarding tenants gradually as soon as the rollout is complete. For those of you interested on getting your tenant onboarded to be able to receive this fix sooner, please fill this survey. We thank you all for the patience! 📎

Already full?

You don't have permission to view or respond to this form

Technical details
Session id: cca0d6c4-cb08-4503-93ac-e1347b677bf1, Correlation id: 19aad79e-875d-4904-9dc4-b6abd26c53fa

[azcloudfarmer]

Hello Everyone - we are excited to share the deployment of this fix is in progress and we will begin onboarding tenants gradually as soon as the rollout is complete. For those of you interested on getting your tenant onboarded to be able to receive this fix sooner, please fill this survey. We thank you all for the patience! 📎

Already full?

You don't have permission to view or respond to this form

Technical details
Session id: cca0d6c4-cb08-4503-93ac-e1347b677bf1, Correlation id: 19aad79e-875d-4904-9dc4-b6abd26c53fa

@jikuja - thank you for catching this. The form is now open to all.

[MarkFarkas]

Hello @azcloudfarmer, can you please provide an update on the status of the rollout? When can we expect all tenants to be onboarded?

[alex-frankel]

We are blocked on rollout at the moment due to a planned end-of-year deployment freeze for specific high-profile events. Most likely we will only be able to onboard the new tenants in specific regions during November, then all regions in early December.

Apologies, as we were not as plugged into the deployment freeze timeline as we should have been. We very much want to get this out ASAP :(

[alexrecuenco]

Is this rolled out already? How can I verify if my tenant already has this upgrade rolled out?

I filled the survey to try to get it sooner, but I am not sure that ever happened, I am still getting mostly resource ignored and "going blind" :')

[alex-frankel]

Sadly, the Azure-wide deployment freeze got extended through all of January. More than likely we won't be able to roll this out until February. Sorry about the delay, but this one is totally out of our control. Thanks everyone for the collective patience.