Expose YAML validation features as standalone tool

[TingluoHuang]

Looking at this document (which I found with some help) it seems that this repository contains all the necessary functionality to validate YAML pipeline definitions locally, without actually sending any build commands to build servers.

I tried to use run.sh --yaml --what-if as shown in that document, but that complained about not finding bin/Agent.Listener and I couldn't find any instructions on how to build that file correctly. However, based on the descriptions in the document, it seems to do exactly what I would need to avoid having to push code to the repository just to see if my build config is valid.

It would be awesome if that could be exposed as something I can run from within my repository (e.g. a dotnet global tool).

[tomkerkhove]

+1 on this - Preferably via a CLI which we can pull in via NPM and maybe even a Azure DevOps task as well.

[bryanmacfarlane]

Yes, it's on our short backlog to create a cli

[ghost]

Did this ever get built?

[kumarharsh]

Another thing which can be done is to make it like Gitlab - you can create an online validation tool (which is basically just a textarea) in the Azure Pipelines web app. The user will paste their yaml, and it'll show all the validation errors.

[tomkerkhove]

That's a good tool indeed and would certainly help!

However, a standalone tool is a must as it should be able to run as part of a CI build.

[tomkerkhove]

/cc @gopinathch

[stevenbrix]

@kmahone i know you’d appreciate this!!

[adesokanayo]

Is there any progress on providing this tool. The tool should help you validate and test your yaml instead of running your pipelines all the time.

[ihnorton]

Since it has not been mentioned, the VSCode extension does at least some local validation (I get red squiggles upon using invalid keywords or indentation): https://marketplace.visualstudio.com/items?itemName=ms-azure-devops.azure-pipelines

(the extension appears to use ajv, which I tried to run manually, via pajv, but got some errors -- for a pipeline schema which otherwise builds successfully. Probably just a configuration issue. Have not dug further as the extension is working great for me - thanks!)

[vtbassmatt]

We have shipped a server-side API for this. No plans to ship a fully standalone version, though, because the amount of server logic we would have to reimplement is too great.

POST to dev.azure.com/<org>/<project>/_apis/pipelines/<pipelineId>/runs?api-version=5.1-preview with a JSON body like this:

{
  "PreviewRun": true,
  "YamlOverride": "
# your new YAML here, optionally
"
}

The response will contain the rendered YAML.

[tomkerkhove]

Awesome, thanks!

@vtbassmatt Are there plans to provide a task for this at least?

It's a bit cumbersome to re-implement this for every customer.

[vtbassmatt]

No plans to implement a task. By default, the build identity doesn't have permissions to queue a dry-run build. But if you alter that, it's a relatively short shell script:

pool:
  vmImage: ubuntu-latest

steps:
- checkout: none
- bash: |
    RUNS_URL=${SYSTEM_TEAMFOUNDATIONCOLLECTIONURI}${SYSTEM_TEAMPROJECT}/_apis/pipelines/${SYSTEM_DEFINITIONID}/runs?api-version=5.1-preview
    echo $RUNS_URL
    curl --user PAT:$(System.AccessToken) \
      -sS \
      --header "Content-Type: application/json" \
      --request POST \
      --data '{"PreviewRun": true,"YamlOverride":"steps:[{checkout:none}]"}' \
      $RUNS_URL

(Obviously replacing the payload with your chosen YAML.)

[tomkerkhove]

Thanks!

[JustinGrote]

@vtbassmatt This works great, I made a powershell cmdlet for it.
https://gist.github.com/JustinGrote/4ece827540da00d5ffce1d9eb593e56c

One thing: The most useful thing for this would be testing templates to make sure they expand properly. Is there any way in the YamlOverride to specify multiple files or somehow test templates?

[vtbassmatt]

@JustinGrote that cmdlet is great, thanks for sharing! We don't have a multi-file version of this endpoint right now, but feature request noted.

[JustinGrote]

@vtbassmatt noted. What I would recommend is that since yaml supports multiple document syntax with '---', to be able to add a comment or something to indicate which file it would be (by file path), that way the document and all its related sub templates and whatnot could be supplied as one entry to YamlOverride

[tomkerkhove]

@JustinGrote Any plans on publishing this to PowerShell modules gallery?

+1 on multi-file support as we heavily use YAML templates

[jmezach]

We are also using templates heavily, but have centralised them in a single repository. I've now setup a build pipeline for that repository to validate all the YAML pipelines that use that repository with the new changes being made to it. This allows us to validate the changes before merging them which could potentially break all our pipelines. Not anymore with this feature which is nice. Maybe I should write a blog post ;).

[tomkerkhove]

If YAML templates are supported then I'm happy - Will give it a go.

Feel free to write that post, happy to share.

[JustinGrote]

@tomkerkhove it looks like @SebastianSchuetze is going to add it to the https://github.com/DarqueWarrior/vsteam module based on a twitter exchange we had :)

[tomkerkhove]

Great to hear, thanks!

[SebastianSchuetze]

@tomkerkhove I am currently sitting at the code for the PR

@JustinGrote do you have a direct link to the API documentation. It's kinda hard to find for me.

[JustinGrote]

@SebastianSchuetze it's part of the preview API docs but I can't find the link anymore, however both of these reference the info:

#34 (comment)
https://docs.microsoft.com/en-us/azure/devops/release-notes/2020/pipelines/sprint-165-update#preview-fully-parsed-yaml-document-without-running-the-pipeline

Also if you look at my source code it's laid out there too.

[vtbassmatt]

We're working on getting the API docs to accurately reflect this (and other Pipelines endpoints).

[JustinGrote]

@vtbassmatt thank you, and FYI while this is a welcome change, the inability to resolve/test templates (which requires a separate file unless theres some magic you can bestow upon me) kinda limits its usefulness, since expanding ${{ each and ${{ if statements is my primary big use case for this vs. commit over-and-over.

[stevenbrix]

So I've been testing this out and this seems to only work if your entire pipeline definition is in a single file, and/or you are only trying to validate the pipeline entry point file, which doesn't work for us. We generally use templates, and as soon as you do that, it doesn't seem like it's possible for those to be picked up. If I make a typo in my local template file and then run this script, it succeeds and I see that my local files aren't being used in the expanded yaml. Is this scenario not supported?

[JustinGrote]

So I've been testing this out and this seems to only work if your entire pipeline definition is in a single file, and/or you are only trying to validate the pipeline entry point file, which doesn't work for us. We generally use templates, and as soon as you do that, it doesn't seem like it's possible for those to be picked up. If I make a typo in my local template file and then run this script, it succeeds and I see that my local files aren't being used in the expanded yaml. Is this scenario not supported?

@vtbassmatt already commented on this. the short answer is "not yet"

#34 (comment)

[stevenbrix]

@JustinGrote - thanks! I missed that (somehow)

[joergjo]

Has something changed very recently? I've been trying this API for the first time today, and neither curl nor the PowerShell function provided by @JustinGrote work for me.

• Authorization for this API (not other DevOps APIs) only works if the username part of the HTTP Basic creds is left blank.
• The response is a redirect to a web page. If run through a browser, I end up at https://aex.dev.azure.com/me?mkt=en-US

[vtbassmatt]

@joergjo that sounds like an authentication or authorization problem; nothing in our API would trigger that behavior. You can call other APIs without a problem, though?

[JustinGrote]

FYI, related pull request to add this to the VSTeam Powershell Module
MethodsAndPractices/vsteam#272

[4c74356b41]

what scopes do I need on the PAT for this to work? the bare minimum, obviously. thanks

[vtbassmatt]

@4c74356b41 Build - read & execute