Improve CTQ notifier

[chrisklus]

This issue is for the remaining work to do for CTQ notifier:

• convert QuickServer.js to TypeScript no-op, QuickServer is node code
• better commenting and small cleanup in QuickServer - completed in CTQ status is not reliably reported to Slack#continouous-testing #166
• investigate to see if there is a way to wait for a complete push to multiple repos before running a CTQ loop that sometimes reports false positives
• investigate if there is a better way to handle a high load of errors, as described by @zepumph in Report all new CTQ errors on Slack #141 (comment)

[chrisklus]

@samreid reported in phetsims/chipper#1269 (comment) an idea for a better way to notify CTQ that a set of pushes is complete! This would address the third checkbox in the list above.

[samreid]

The same idea could be leveraged for the full CT like so:

• The push script notifies that it is about to push certain repos. fetch('http://bayes.colorado.edu/ct/prepush?repos=sim1,sim2,common1)
• The push script pushes the repos
• The push script notifies that it has pushed the repos. fetch('http://bayes.colorado.edu/ct/postpush?repos=sim1,sim2,common1)

Then CT and CTQ can identify whether they are in a consistent state if they aren't waiting for any prepushes to finish. Or maybe we would just use a random token like start=n2th1nth23n12th3n and end=n2th1nth23n12th3n. CT could give up if it doesn't receive the end for a start within N minutes.

[pixelzoom]

In Slack#continuous-testing, I suggested something else:

It seems like many (most?) of the CTQ false positives (aka 🦤) occur when pushing to multiple repos. I don’t know how CT knows when to pull repos. But I suspect it notices that a repo has changed, then pulls only that repo. Could this problem be resolved (or reduced in frequency) by introducing a slight delay after CT notices any repo change, then doing a pull-all?

[pixelzoom]

In Slack#continuous-testing, @zepumph said:

The logic is here, it would be simple enough to instead wait 20 seconds before running it.

aqua/js/server/QuickServer.js

Lines 111 to 121 in a2dcc1f

	// TODO: don't be synchronous here
	const staleRepos = [];
	winston.info( 'QuickServer: checking stale' );
	await Promise.all( reposToCheck.map( async repo => {
	if ( await isStale( repo ) ) {
	staleRepos.push( repo );
	winston.info( `QuickServer: ${repo} stale` );
	}
	} ) );
	if ( staleRepos.length || forceTests ) {

[zepumph]

@chrisklus, 1de7fdf was causing a bunch of problems because it was kicking off infinite numbers of setTimeout calls and running out of memory on the javascript heap. We need to synchronously pause execution of the while loop if we find a staleRepo. Let's try this one.

[zepumph]

Working on this today as part of #166

[chrisklus]

@zepumph do you think we can close this issue? Or wait another month or so to see if item 3 in #152 (comment) needs more work?

[zepumph]

I was just thinking that! I do not think any more is needed here. Thanks