[Feature] End-to-end integration tests

[arcanis]

I'm working to integrate a basic daily cron job that would test Yarn with various projects from the ecosystem (Gatsby, Webpack, ESLint, Babel, Jest, etc), in an effort to quickly detect regressions.

I've wrote the pipeline in azure-e2e.yml, but I'm not quite sure how to integrate it with GitHub. In particular, I'd like to open an issue for each job that didn't work and for which a matching issue doesn't already exist (we can use @yarnbot for that). I could use the GitHub API, but I'm not quite sure how to access the status of the jobs from within the Azure Pipeline itself. Does someone have an idea?

cc @zenithworks, I remember you offered to help us integrate the changelog into Azure on the v1 repository, is that also something in your area of expertise? It would be great if Azure could help us validate the sanity of the JS ecosystem 😄

[arcanis]

Should be ok! I've hacked my way around: each acceptance test goes through a report.sh indirection, which adds a new entry in a JSON file. This JSON file is then parsed once all the tests have been run, and voilà ✌️

The current tests should run on Gatsby and Create-React-App. I've tried the report system (opening / closing GitHub issues when the problems are detected then resolved) in a vaccum and it seemed to work!