Auto-run Selenium on release

[gsnedders]

The only real challenge here is, AFAIK, the total runtime? Otherwise one could quite easily have a GitHub Action that did this? Could one reasonably split it up into smaller jobs and use concurrency to limit it to running one job at a time (to avoid exceeding BrowserStack/Sauce/etc. limits)?

[foolip]

Until recently, #1230 was my main concern, that I didn't entirely trust that we'd get complete and accurate data.

One remaining concern I have is #1199, where mapping of browser versions on the part of BrowserStack or Sauce might cause us to either redundantly test the same config twice. I don't think we could end up with mixed up versions, but I'm worried if that's even a remote possibility.

But all together, yes this seems doable!

The simplest way to do this is probably a job that runs right after the deploy job, and then creates a PR for https://github.com/foolip/mdn-bcd-results. Maybe it should also delete all existing results.

@vinyldarkscratch WDYT, are there additional hurdles?

[queengooborg]

There's a few other issues we'll need to take care of before an auto-run of the Selenium script is ready to go, specifically if we want to run it with little to no gaps in the generated data.

The first is #1281, which I still haven't been able to get to the bottom of (though admittedly I haven't really spent any time on it). Since these browser versions are freezing, we're unable to automatically collect results for them.

The other is an issue regarding mic/camera permissions on macOS Catalina and Big Sur with BrowserStack, which is preventing the automatic collection of Firefox results for macOS. I'm in communication with BrowserStack to resolve the issue. (This one, though, we might be able to resolve by preferring SauceLabs!)

[foolip]

It sounds like those issues might not be blockers for collecting a more modest set of data. Would we be able to get the following without issue?

• Past 2 years of Chrome, Edge and Firefox releases on Window 10
• Past 2 years of Safari releases on macOS

[queengooborg]

If we limit to just the past two years of browser releases and aren't worrying about Firefox on macOS, then yes, we can avoid those issues!

[foolip]

Alright, great! At this point this is "patches welcome", probably pretty easy but will require a whole lot of iteration and waiting for CI.

[gsnedders]

Honestly I'd rather more than just the past two years; half my desire here is wanting data for (much) older releases to hand.

I'd forgotten we actually ran each browser version once per OS; we could presumably cut down what we run in CI by running each browser version only once per OS, especially/at least until #571 is dealt with.

[foolip]

Reposted at openwebdocs/mdn-bcd-collector#725.