Create local UI editor for matrix generator

[benbp]

This would help the job matrix editing experience. It would allow people unfamiliar with the matrix to see how different fields behave, and could also make visualizing the output matrix easier.

Another possibility could be a vscode plugin and/or codespaces support.

[weshaggard]

While not opposed to a UI it feels like that might be more work then we need. Don't we already have some test scripts people can run to see what the output will be? Maybe we invest in make those a little easier to use and provide debugging info.

[benbp]

Perhaps the discoverability is a factor. @vhvb1989 do you have more thoughts?

Related discussion: Azure/azure-sdk-for-cpp#3553 (comment)

[kurtzeborn]

This would be a good candidate for an intern project.

[vhvb1989]

Don't we already have some test scripts people can run to see what the output will be?

That would be at least a very good start.

If the EngSys team expect people within each language-team to be able to make changes to the matrix configuration without any EngSys's human help, I would expect a high investment in documentation, guide and tools for people to learn by themselves. I bet there is currently some available documentation, and it might be enough to add the link to the docs from the pipeline yaml files.

But, if teams should rely on EngSys support to make any changes within the /eng folder, then we don't really need this.

IMO, the matrix generator is a magnificent piece of art <3, and it has the potential to become a very well appreciated tool for any project using ADO. Right now we are calling the archetype-sdk-tests-generate.yml template from /eng/common and passing the matrix.json configuration, but it could become an ADO pipeline task directly that takes the matrix json. So, going into that direction, a GUI to visualize the output (even a simple table), would be lovely :)

[benbp]

@vhvb1989 yes I think we need to add more links in the yaml for docs (I have just filed this issue).

The current matrix documentation is here: https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/scripts/job-matrix/README.md

Reading through, I'm realizing that while there are examples of how to run the generator for local development (e.g. here). There is no dedicated section in the TOC for "Local Testing and Development" or something like that. I will add that section as part of this issue. For example, in the cpp repo:

# powershell
./eng/common/scripts/job-matrix/Create-JobMatrix.ps1 `
    -ConfigPath eng/pipelines/templates/stages/platform-matrix-live.json `
    -Selection all  # or "sparse"

The generated output for the pipeline run is also printed if you click into the generate_matrix_* job, e.g. here.

[benbp]

I don't think we will plan to invest much more in the ADO task area, but as we move more and more functionality into github actions, I could see us making the matrix generation components a standalone github action. The matrix generation config was inspired by the github actions matrix originally, and we started by extending that model with sparse support, then later adding filters, replace, etc.