Karma BrowserStack sessions hanging and failing for no reason

[Finesse]

This is what it looks like:

(in fact the devices ran successfully)

(the timed-out session are retries, and there was no reason to retry, because the first attempt succeeded)

These problems are caused by the same reason. They happen with iOS often and with Chrome 65 on Windows 10 sometimes. They don’t always cause the testing process to fail, but they occupy the BrowserStack Automate quota for a long time.

These problems have the same reason. This is what happens under the hood:

1. In the first attempt the Karma launcher gets to the point where it tells the browser to open the Karma’s page. The promise doesn’t resolve despite the browser actually opening the page and running the tests successfully. Before that, the capture timeout starts.
2. When the tests finish, Karma kills the browser triggering its kill handler. It causes the BrowserStack session to finish normally. By design, Karma must stop using this launcher at this moment, but…
3. Eventually the “open the Karma’s page” promise rejects. That causes the red message from the first screenshot (reminder: actually the browser opened the page and ran the tests successfully). That also calls done() with an error, which triggers a launcher restart (starting the regular launcher lifecycle again on the same launcher object). The 2nd starts creating a BrowserStack session.
4. Shortly after the 2nd attempt start (before the BrowserStack session is fully created), the capture timeout of the first attempt finishes. The timeout kills the launcher (because it’s in the “being captured” state). Maybe the capture timeout wouldn’t trigger if the original author didn’t forget to save the timeout id.
5. I’m not sure what happens next. Probably, Karma thinks the launcher is dead (not doing anything), so it exits because the actual tests have run, meanwhile the BrowserStack session keeps handing without commands. Anyway, killing the launcher preliminary because of a leftover from the previous attempt is not right.

Therefore the potential solution should:

• Isolate the attempts in closures. When an attempt is “killed”, it must not print error messages and call any launcher method. Also the browser instance must be a part of the closure so that it can be stopped when the attempt is not current.
• Fix the capture timeout.
• Decrease the BrowserStack session idle timeout (current 5 minutes is too much).