Add continuous testing for other browsers

[samreid]

At today's meeting, we discussed the fact that Bayes CT is only reporting testing on one version of Chrome. We would like to extend this to cover other browsers, to catch issues like the ones that came in phetsims/tasks#932

We discussed getting another Windows machine (not a server) that could run testing in several browsers (including IE, firefox, edge, chrome).

Assigned to @ariel-phet to discuss hardware, and to @jonathanolson to discuss if any changes need to be made to CT to know what browser was reporting.

[ariel-phet]

Basically, a device just needs to open a particular web address, nothing needs to be loaded locally on the test device, except the web browsers.

So ideally a device where you can have multiple tabs of the same browser open (CT chooses a random test of available browser tests and tries to pick ones with the lowest "count" of tests). CT server is responsible for making the appropriate snapshots and builds. Ideally also multiple browsers.

Probably want the device on Ethernet (with quality network card). Currently we are only testing chrome, would be ideal to expand to other browsers in list of priority:

1. IE 11
2. Edge
3. Mobile Safari
4. Firefox
5. Safari

Currently CT is testing 10 tabs (which is putting a fair bit of load on Bayes). We could reduce this load (maybe just have 5 tabs open on Bayes) if we offload some of the other work to desktop laptops.

Easiest to have 2 separate devices probably (one windows, one mac). Could potentially set up some virtual machines (could discuss this with OIT). Sims should generally not be using more than 100-200 MB of RAM (so 20 sims would be fine with 4 GB), so memory probably not the bottleneck more CPU. So spend money on CPU not memory at first. Multi-core CPU's would be best (this type of task benefits from parallelism). So go for more cores over more speed per processor.

Also want to optimize download speeds.

@jonathanolson says we will need to add "user agent" to CT so we can at least know what browser things broke on.

[ariel-phet]

After looking around for multi-core solutions and mainly seeing very expensive workstations, I spoke with JR at JILA. It seems two easy routes to go would be a refurbished workstation (affordable, but likely overkill in many areas), or as custom build.

JR says:

AMD's processors are great for multiple cores at the moment. They've got an 8-core chip that's pretty reasonable. Here's a decent build: https://pcpartpicker.com/list/hMBZJ8 -- could go cheaper on the RAM by a little. I'd definitely recommend the SSD though... it'll just keep the machine feeling snappy. It'd be a pretty good build though!

This build is pretty dirty cheap. That link has a compatibility warning, but JR said the following:

That motherboard can definitely support that processor per Asus' spec sheet. If it doesn't support it out of the box, then I'll order one for testing to have at JILA that we can use to update the BIOS for you. I can even just order that to have it, and if we don't need it, I can build a machine out of it for someone else later.

He also claimed it would only take a couple of hours of his time to put this together. I will discuss with @oliver-phet - but at this pricepoint it seems well worth exploring this solution.

[samreid]

My notes from slack: It would probably be better to get a laptop instead of a desktop. Smaller footprint, easier to move, always has a screen and keyboard, etc, inherent battery backup. The only reason I would recommend a desktop is if we needed something cheaper or easily upgradeable.

Additionally, a laptop is likely to be quieter, which may be a consideration if the machine is in one of our offices. I'm bringing this up now because we had a prior experience at PhET where we needed a machine for a similar service and went with a desktop and it only took us a few days before we wished it had been a laptop.

[ariel-phet]

@samreid your concerns on a laptop are noted, but this machine does not need to live in an office necessarily (could possibly go in our storage closet or such). In terms of the functional requirements for the machine, getting an appropriate/equivalent laptop would likely be exponentially more expensive.

[ariel-phet]

@oliver-phet suggested re-purposing Simian, we are going to explore that possibility.

[oliver-phet]

Are there any storage concerns with this machine? Simian has 5x 2TB drives that I could raid, but wanted to make sure there aren't any particular requirements (e.g. multiple partitions with different operating systems).

[oliver-phet]

@ariel-phet Do we want to save any of the data that is currently on the hdd's?

[ariel-phet]

@oliver-phet I don't think any particular storage concerns.

As for the data, all this data has been backed up to the PetaLibrary (and that backup has been verified multiple times).

So we have full permission to wipe these drives.

[oliver-phet]

Simian has been wiped and Windows 10 is installed. It's currently living on Arnab's desk in my office, but once it's ready to be used I can move it back down to the 2nd floor interview room.

[oliver-phet]

An update on this issue: I'm waiting on Veronica Lingo to complete the static IP address request.

[oliver-phet]

Static IP address should be assigned. I plugged in the server (in F925) and it connected immediately.
@jonathanolson I think that's everything that you were waiting on? Let me know if there is anything else I can help with on this.

[jonathanolson]

Do we have the remote desktop setup (and if so, where are the credentials? Google Docs?) I'm presuming it will need VPN to connect if off-campus.

I'm interested in opening browser tabs (for whatever browsers we want) pointed to https://bayes.colorado.edu/continuous-testing/aqua/html/continuous-loop.html?id={{ID_HERE}} where {{ID_HERE}} is replaced with an identifier for the device+browser (and encoded, so spaces are %20). For example, I was using https://bayes.colorado.edu/continuous-testing/aqua/html/continuous-loop.html?id=JO%20Win10%20Chrome when I was running Chrome on my home computer.

[oliver-phet]

I added the remote desktop information to the credential doc (may be at the bottom). I'm not sure if you'll need additional information to connect via VPN.

[ghost]

@ariel-phet, @jonathanolson, Simian's fan(s) rev up every once in a while, so I assume that means Simian is working. Can this issue be closed? Anything else that needs to be done?

[zepumph]

Over in phetsims/perennial#153 we decided it would be really nice to have IE testing built sims on CT. Pinging here for potential priority boost!

[zepumph]

Another ping here because of phetsims/energy-forms-and-changes#320. It would be really nice if we could know when changes break IE well before we get to the QA process. Adding to developer meeting to see how feasible adding IE to CT would be.

[jessegreenberg]

Was discussed on 5/21/20 -

Issues in #80 (comment) and #80 (comment) were related to IE11, maybe not as important if we decide to drop that platform that often gives us issues.

[ariel-phet]

Considering that making this improvement is not really a priority, and that IE11 has been dropped, I feel like we can close this issue.