Add Repeat and Retry functionality to sigh test

[thebrianchen]

Adding the option to allow retries and repeat tests. I initially tried adding repeat functionality directly into mocha by encapsulating runner.run() which would allow test pass/fail/end hooks to be added more easily. However, I ran into a mocha implementation issue that requires the entire mocha instance to be re-instantiated and the 'require' cache cleared in between runs.

I removed the return from test() -- not sure where it's being used.

Also, not sure if the package.lock should be excluded // how to auto-exclude it from future commits.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here (e.g. I signed it!) and we'll verify it.

---

What to do if you already signed the CLA

Individual signers

• It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

• Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
• The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
• The email used to register you as an authorized contributor must also be attached to your GitHub account.

[thebrianchen]

I signed it!

[googlebot]

CLAs look good, thanks!

[lindner]

Check the return-code you're generating? If any test fails you should return non-zero for the test phase.

😱
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] test-sigh: `cross-env tools/sigh`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] test-sigh script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm ERR! A complete log of this run can be found in:
npm ERR!     /home/travis/.npm/_logs/2019-01-15T21_18_55_679Z-debug.log

[lindner]

LGTM, Raul for approval.

[raulverag]

I don't see why we would want to repeat or retry tests within one run. What does it even mean if the tests pass on one repeat and then fail on another, other than that the tests are flaky? Furthermore, it seems that if the last run succeeds, the user won't even see any failing output, because that will have scrolled off the screen. Correct me if I'm mistaken, and please elaborate as to what purpose repeats and retries serve.

[shans]

Friendly reminder to please discuss changes (such as this) which have the potential to change development practices for all of us in a team meeting, ideally before starting development.

[lindner]

For folks tuning in, this work was done to specifically address issue #779

My apologies for not bringing this up in a larger context. It was deemed something simple enough as a good-first-issue.

[lindner]

Raul suggested that this needs to have summary output at the end.

[raulverag]

Yes. If the idea is to help diagnose flaky tests, as described in #779, then the results must be displayed in some kind of summary, e.g. "x out of y tests succeeded. First failure was repeat N.", or in the case of retries, "Tests succeeded after X retries".

[thebrianchen]

The mocha.js implementation of retries does not allow for logging and will simply run the test until it either passes or the retry count is exhausted, so I removed the the retries flag option.

Added a summary to the end of the runs. Example output:

100 runs completed. 4 runs failed.
Failed runs: [4, 47, 49, 81]

[lindner]

LGTM, Raul for final approvals.