Feature: Support validation and parameter intellisense for linked templates

[StephenWeatherford]

NOTE: The spec info in this bug is out of date, but the feature is mostly implemented.
Remaining work to consider for later has been moved to #1211

P2

• ISSUE Keep it from being closed by vscode after 3 min?? – errors will disappear in the linked template after 3 min
• Support linked parameter files -> Linked template support improvements #1211
• UTF-8 paths -> Linked template support improvements #1211
• Spaces in paths
• Path case insensitivity
• ISSUE: Linked parameters don't get validated (and we don't show linked template info) if there are any other validation errors
• Show linked templates in ARM Template Outline -> Linked template support improvements #1211
• ISSUE: COPY loops show multiple times in UI
• Light bulb: Add missing parameters when there is no "parameters" object in the deployment resource -> Linked template support improvements #1211

Later

• Open linked template file with the actual set of parameter values that would be passed in -> Linked template support improvements #1211

TODO: Update this section

Spec notes

In scope: literal URIs and relative URIs

• Tooling for the relative path experience will provide the most customer value and is the scenario that we should optimize for

Dependencies

Validation assembly

#745 Validation API changes to support linked template validation

Extension work

#1116 Linked Templates phase 1

Spec Issues

• Need relative path spec from @alexander Frankel and availability date
• Won't find any linked templates if full validation not enabled
  • Turn on full validation if all parameters have values We should enable full validation if all parameters in the template have default values (even if there's no param file) #512
  • Show information diagnostic if full validation is off
  • CUT Need a way to provide secureString / secureObject parameter values to unblock full template validation #704 - provide dummy value if appropriate
• Won't find second linked template if first has an error
• What's the intellisense/rename/completion experience inside a linked template as far as parameters/outputs go?
  • e.g. if you rename a parameter in a linked template, does it rename it in the parent template's deployment for that linked template? -> Linked template support improvements #1211
• Editing experience for outputs? No - that would be an experience for the reference() function's return value
• What's the experience when a linked path can't be found?
  • A: Place an error in the Issues list
• Re-evaluate when linked template file changes - how?
• Handle deployment()? Right now we return http://fake
  • eg "uri": "[uri(deployment().properties.templateLink.uri, 'static-public-ip.json')]"
    • cannot open https://fake/static-public-ip.json. Detail: Unable to resolve resource https://fake/static-public-ip.json
• How show progress during load phase?

Template URI:
We will handle both relative and absolute URIs

Validation:

Download and buffer template (do we open in tab or not?)
If required parameters are defined in template A, or a parametersLink exist, provide full validation on buffered template (can we do the first part of this?).
Provide schema and expression validation on buffered template.
Error message with line and location from buffered template but provide no navigation to the location.

Authoring Experience:

Template A parameter completion for template B. Can we also identify required vs optional in the completion experience?
Template A output completion from template B

Relative Path:

Validation:

Open template B from specified path
If required parameters are defined in template A, or a parametersLink exist, provide full validation on template B (can we do the first part of this?).
Provide schema and expression validation on template B.
Error message with line and location from buffered template and potential to provide navigation to the location.

Authoring Experience:

Same as with Template URI.

Other thoughts:

Could we do an insert linked template feature which would take in URI / Path and scaffold out the deployment resource, target. and parameters?
Investigate how this will work when we have linked templates with a depth greater than 2 (A > B > C)
When working with relative / on disk templates is there anything to consider if template B has a parameter file mapping?

[ishepherd]

Will relative URIs support this? https://feedback.azure.com/forums/281804-azure-resource-manager/suggestions/17275646-enable-using-local-filesystem-for-linked-templates

(🤞 🤞 🤞 🤞 )

[alex-frankel]

We are going to support a new property for nested deployments called relativePath to help support this.

The first place we are going to support relativePath is for template specs, which we discussed in some depth at //build for the first time.

For generic deployments, in the short term, we are going to treat relativePath as syntactic sugar for the uri pattern for generic deployments that many customers use today. However, you will still need to stage the files yourself to a public location. At some point, we are going to support packaging up deployments w/ relativePath in the same way that we will be packaging up templateSpecs using the CLI tools, so that you won't need to worry about staging.

VS code will understand and support the relativePath property as part of this work, but it won't be in the first phase.

[neilpeterson]

@StephenWeatherford some notes on the relative path implementation.

Implementation

Adding support for a relativePath field to the TemplateLink object of the Microsoft.Resources/deployments resource type.

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "resources": [
        {
            "name": "nestedDeployment1",
            "type": "Microsoft.Resources/deployments",
            "apiVersion": "2019-10-01",
            "properties": {
                "mode": "Incremental",
                "templateLink": {
                    "relativePath": "./assets/azuredeploy.json"
                }
            }
        }
    ]
}

Relative Path fields should always use the local file as the starting point (rather than the root template). The customer can specify this implicitly, or with a prefix like ‘.\’ or ‘\’. The relative path will also support / and \ as folder delimiters.

Examples

Example directory / file structure:

• TopLevel\entryTemplate.json ** this is the top-level/parent template for all other deployments
• TopLevel\commonTemplate.json
• TopLevel\nestedDir\nestedTemplate.json
• TopLevel\nestedDir\doubleNesting\doubleNestedTemplate.json
• inaccessibleTemplate.json

From entryTempalte.json, commonTemplate.json can be deployed in the following ways:

"TemplateLink": { 
    "relativePath": "commonTemplate.json", 
} 
 
"TemplateLink": { 
    "relativePath": "./commonTemplate.json", 
} 
 
"TemplateLink": { 
    "relativePath": "\commonTemplate.json", 
} 

From entryTempalte.json, nestedTemplate.json can be deployed in the following ways:

"TemplateLink": { 
    "relativePath": " nestedDir\nestedTemplate.json ", 
} 

"TemplateLink": { 
    "relativePath": " ./nestedDir/nestedTemplate.json", 
}

From nestedTemplate.json, the other two files can be deployed like this:

"TemplateLink": { 
    "relativePath": " ..\commonTemplate.json", 
} 

"TemplateLink": { 
    "relativePath": " ./doubleNesting/doubleNestedTemplate.json", 
} 

Finally, inaccessibleTemplate.json will not deploy properly, as it is outside the directory of the parent template.

Other considerations

One of the following will be required in the TemplateLink object, however due to expressiveness limitation of JSON schems, this will not be enforced by a schema.

"TemplateLink": { 
    "uri": <string>, 
    "id": <string>, 
    "relativePath": <string>, 
    "contentVersion": <string>, 
} 

If for example, both uri and relativePath are specified, the deployment is invalid.

The validation API will catch this on pre-flight. We will need to consider how / where to put a similar check for the extension. We need to considering bringing this code into the extensions validation code.

[jdelforno]

This will be extremely helpful considering GitHub Actions allows the repo to be checked out as is and execute the ARM Templates as a local resource within the Runner.

Thanks for your efforts, please let me know if you'd like testing done.

[StephenWeatherford]

@jdelforno Great to know, thanks! Probably starting on this next wek.

[StephenWeatherford]

Just released 0.13.0, starting to look at this. Will probably come in phases, the intention is for this to be milestone 0.15.0, but there is a 0.14.0 release planned which will be some minor features and some bug fixes, with 0.15.0 starting in parallel.

Are there any public repos with examples of how you guys have things set up? The intention was to start with only supporting relative paths (the new relativePath), which is not in public preview yet, but obviously you guys are not using that yet.

[jellebens]

It makes sence that when using devops u publish your arm scripts to a dropfolder as an artifact and deploy them from there without having to upload to a blobstorage first

[mkulisic]

@StephenWeatherford some notes on the relative path implementation.

Implementation

Adding support for a relativePath field to the TemplateLink object of the Microsoft.Resources/deployments resource type.

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "resources": [
        {
            "name": "nestedDeployment1",
            "type": "Microsoft.Resources/deployments",
            "apiVersion": "2019-10-01",
            "properties": {
                "mode": "Incremental",
                "templateLink": {
                    "relativePath": "./assets/azuredeploy.json"
                }
            }
        }
    ]
}

Relative Path fields should always use the local file as the starting point (rather than the root template). The customer can specify this implicitly, or with a prefix like ‘.\’ or ‘\’. The relative path will also support / and \ as folder delimiters.

Examples

Example directory / file structure:

• TopLevel\entryTemplate.json ** this is the top-level/parent template for all other deployments
• TopLevel\commonTemplate.json
• TopLevel\nestedDir\nestedTemplate.json
• TopLevel\nestedDir\doubleNesting\doubleNestedTemplate.json
• inaccessibleTemplate.json

From entryTempalte.json, commonTemplate.json can be deployed in the following ways:

"TemplateLink": { 
    "relativePath": "commonTemplate.json", 
} 
 
"TemplateLink": { 
    "relativePath": "./commonTemplate.json", 
} 
 
"TemplateLink": { 
    "relativePath": "\commonTemplate.json", 
} 

From entryTempalte.json, nestedTemplate.json can be deployed in the following ways:

"TemplateLink": { 
    "relativePath": " nestedDir\nestedTemplate.json ", 
} 

"TemplateLink": { 
    "relativePath": " ./nestedDir/nestedTemplate.json", 
}

From nestedTemplate.json, the other two files can be deployed like this:

"TemplateLink": { 
    "relativePath": " ..\commonTemplate.json", 
} 

"TemplateLink": { 
    "relativePath": " ./doubleNesting/doubleNestedTemplate.json", 
} 

Finally, inaccessibleTemplate.json will not deploy properly, as it is outside the directory of the parent template.

Other considerations

One of the following will be required in the TemplateLink object, however due to expressiveness limitation of JSON schems, this will not be enforced by a schema.

"TemplateLink": { 
    "uri": <string>, 
    "id": <string>, 
    "relativePath": <string>, 
    "contentVersion": <string>, 
} 

If for example, both uri and relativePath are specified, the deployment is invalid.

The validation API will catch this on pre-flight. We will need to consider how / where to put a similar check for the extension. We need to considering bringing this code into the extensions validation code.

Hello,
I have been trying to use the relativePath on my deployment, but I am running into the following error in my release pipeline:

"Error Reason: Error: Code=InvalidDeployment; Message=The deployment definition is invalid; TemplateLink RelativePath can only be used with api-version '2020-06-01' or later."

Here is the resource I am trying to deploy (I removed the parameters for privacy):

{ "type": "Microsoft.Resources/deployments", "apiVersion": "2020-06-01", "name": "[concat(parameters('ProjectName'),'.ADF.', parameters('Environment'))]", "subscriptionId": "[parameters('subscriptionId')]", "resourceGroup": "[parameters('ResourceGroupName')]", "properties": { "mode": "Incremental", "expressionEvaluationOptions": { "scope": "inner" }, "templateLink": { "relativePath": "[parameters('ADFTemplateFile')]" } } }

I am using the correct version of the API, shouldn't this work?

[StephenWeatherford]

@mkulisic The new relative path property isn't available publicly yet, it's still in private preview. cc @alex-frankel

[stefan-schweiger]

@StephenWeatherford has there been any update on this? I'm waiting desperately for this 😄

[StephenWeatherford]

Been actively working on it (currently OOF). Hopefully will have something out by end of January.

[StephenWeatherford]

See #1116 for first phase.

[DougalKennedy]

@StephenWeatherford do you have a link for applying to join the private preview?

[StephenWeatherford]

@DougalKennedy I'll be sure to let you know when we have something to test, thanks!

[MarcusFelling]

@jdelforno @stefan-schweiger @DougalKennedy We have a new release of ARM Tools for VS Code, that now supports linked template validation. Want to take it for a test drive? You can find the new release here: https://github.com/microsoft/vscode-azurearmtools/releases/tag/v0.15.0-alpha1

We would love your feedback!

[StephenWeatherford]

@jdelforno @stefan-schweiger @DougalKennedy I've uploaded alpha2, which supports the "uri" property (alpha1 only supported relativePath). You can find the release and instructions here: https://github.com/microsoft/vscode-azurearmtools/releases/tag/v0.15.0-alpha2

Don't hesitate to post in our issues with comments or questions, thanks! I hope to release this next week.

NOTE: This invitation is open to anyone.

[StephenWeatherford]

@jdelforno @stefan-schweiger @DougalKennedy
Alpha 3 (and release candidate): https://github.com/microsoft/vscode-azurearmtools/releases/tag/v0.15.0-alpha3

[StephenWeatherford]

Fixed in next release. Possibly Wednesday.

[StephenWeatherford]

This was released on Monday. Let us know if you have problems os suggestions, thanks!